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Accountability  *  Integrity  *  Reliability 


United  States  General  Accounting  Office 
Washington,  D.C.  20548 


March  30,  2001 

The  Honorable  James  M.  Inhofe 
Chairman 

The  Honorable  Daniel  K.  Akaka 
Ranking  Member 
Subcommittee  on  Readiness  and 
Management  Support 
Committee  on  Armed  Services 
United  States  Senate 

With  an  annual  information  technology  budget  of  about  $20  billion,  and 
tens  of  billions  more  budgeted  for  technology  embedded  in  sophisticated 
weaponry,  the  Department  of  Defense  (DOD)  relies  heavily  on  software- 
intensive  systems  to  support  military  operations  and  associated  business 
functions,  such  as  logistics,  personnel,  and  financial  management.  One 
important  determinant  of  the  quality  of  these  systems,  and  thus  DOD’s 
mission  performance,  is  the  quality  of  the  processes  used  to  develop, 
acquire,  and  engineer  them.  Recognizing  the  importance  of  these  processes 
to  producing  systems  that  perform  as  intended  and  meet  cost  and  schedule 
commitments,  successful  public  and  private  organizations  have  adopted 
and  implemented  software/systems  process  improvement  (SPI)  programs.1 


'As  used  in  this  report,  SPI  refers  to  improvements  in  software  development,  software 
acquisition,  and  systems  engineering.  Software  development  refers  to  activities  an 
organization  uses  to  build  and  maintain  software,  while  software  acquisition  refers  to 
activities  an  organization  uses  to  obtain  software  developed  by  another  organization. 
Systems  engineering  refers  to  activities  an  organization  uses  to  define,  develop,  and 
maintain  systems. 
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This  report  is  part  of  our  response  to  your  request  to  compare  and  contrast 
DOD  information  technology  practices  with  leading  practices.  In  particular, 
you  asked  us  to  review  DOD  components’  (military  services  and  Defense 
agencies)  SPI  management  activities  to  ensure  that  DOD  is  taking  the 
necessary  steps  to  continuously  strengthen  its  software  and  systems 
development,  acquisition,  and  engineering  processes.  As  agreed  with  your 
offices,  our  objectives  were  to  (1)  compare  selected  DOD  components’  SPI 
programs  against  Carnegie  Mellon  University’s  Software  Engineering 
Institute’s  (SEI)2  IDEALSM3  model,  which  is  a  recognized  best  practices 
model,  (2)  determine  how  these  components  have  approached 
management  of  their  SPI  programs,  and  (3)  determine  what  DOD-wide 
efforts  are  under  way  to  promote  and  leverage  the  components’  SPI 
programs.  The  components  that  we  selected  were  the  Departments  of  the 
Army,  Air  Force,  and  Navy;  the  Marine  Corps;  the  Defense  Logistics  Agency 
(DLA);  and  the  Defense  Finance  and  Accounting  Service  (DFAS). 

Because  Army,  Navy,  and  Air  Force  do  not  manage  SPI  centrally  and  have 
delegated  SPI  responsibility  to  their  respective  subordinate  organizational 
units,  we  selected  at  least  two  of  the  largest  of  these  units  within  each 
service  to  review.  Accordingly,  all  references  in  this  report  to  the  respective 
services’  SPI  programs  refer  only  to  the  subordinate  units  that  we 
reviewed.  We  performed  our  work  from  March  through  December  2000,  in 
accordance  with  generally  accepted  government  auditing  standards.  (See 
appendix  I  for  details  of  our  objectives,  scope,  and  methodology,  including 
the  specific  service  units  reviewed.)  DOD  provided  us  with  written 
comments  on  a  draft  of  this  report.  These  comments  are  summarized  in  the 
“Agency  Comments  and  Our  Evaluation”  section  of  this  letter  and  are 
reproduced  in  full  in  appendix  II. 


Background 


DOD  maintains  a  force  of  about  3  million  military  and  civilian  personnel 
worldwide.  To  protect  the  security  of  the  United  States,  the  department 
relies  on  a  complex  array  of  computer-dependent  and  mutually  supportive 
organizational  components,  including  the  military  services  and  Defense 
agencies.  It  also  relies  on  a  broad  array  of  computer  systems,  including 


2SEI  is  a  nationally  recognized,  federally  funded  research  and  development  center 
established  at  Carnegie  Mellon  University  to  address  software  engineering  practices. 

3IDEALSM  is  a  service  mark  of  Carnegie  Mellon  University  and  stands  for  initiating, 
diagnosing,  establishing,  acting,  and  leveraging. 
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weapons  systems,  command  and  control  systems,  satellite  systems, 
inventory  management  systems,  financial  systems,  personnel  systems, 
payment  systems,  and  others.  Many  of  these  systems  in  turn  are  connected 
with  systems  operated  by  private  contractors,  other  government  agencies, 
and  international  organizations. 

DOD’s  ability  to  effectively  manage  information  technology  is  critical  to  its 
ability  to  accomplish  its  mission.  Its  reliance  on  software-intensive  systems 
to  support  operations  related  to  intelligence,  surveillance,  security,  and 
sophisticated  weaponry — along  with  financial  management  and  other 
business  functions — will  only  increase  as  the  department  modernizes  and 
responds  to  changes  in  traditional  concepts  of  warfighting. 

The  scope  of  DOD’s  information  technology  inventory  is  vast:  over  1.5 
million  computers,  28,000  systems,  and  10,000  computer  networks.  Further, 
many  of  DOD’s  most  important  technology  projects  continue  to  cost  more 
than  projected,  take  longer  to  produce,  and  deliver  less  than  promised.4  As 
a  result,  we  have  designated  DOD  systems  development  and  modernization 
efforts  as  a  high-risk  area.5 

The  quality  of  the  processes  involved  in  developing,  acquiring,  and 
engineering  software  and  systems  has  a  significant  effect  on  the  quality  of 
the  resulting  products.  Accordingly,  process  improvement  programs  can 
increase  product  quality  and  decrease  product  costs.  Public  and  private 
organizations  have  reported  significant  returns  on  investment  through  such 
process  improvement  programs.  SEI  has  published  reports  of  benefits 
realized  through  process  improvement  programs.  For  example,  SEI 
reported  in  19956  that  a  major  defense  contractor  implemented  a  process 
improvement  program  in  1988  and  by  1995  had  reduced  its  rework  costs 
from  about  40  percent  of  project  cost  to  about  10  percent,  increased  staff 
productivity  by  about  170  percent,  and  reduced  defects  by  about  75 
percent.  According  to  a  1999  SEI  report,7  a  software  development 
contractor  reduced  its  average  deviation  from  estimated  schedule  time 


4  Observations  on  the  Department  of  Defense 's  Fiscal  Year  1999 Performance  Repoit  and 
Fiscal  Year 2001  Performance  Plan  (GAO/NSIAD-00-188R,  June  30,  2000). 

"High-Risk  Series:  An  Update  (GAO/HR-99- 1,  January  1999). 

•Technical  Report  CMU/SEI-95-TR-017,  November  1995. 

Technical  Repoit  CMU/SEI-99-TR-027,  November  1999. 
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from  112  percent  to  5  percent  between  1988  and  1996.  During  the  same 
period,  SEI  reported  that  this  contractor  reduced  its  average  deviation 
from  estimated  cost  from  87  percent  to  minus  4  percent. 

To  aid  organizations  attempting  to  initiate  and  manage  SPI  programs,  SEI 
has  published  a  best  practices  model  called  IDEAL,SM  which  defines  a 
systematic,  five-phase,  continuous  process  improvement  approach,  with  a 
concurrent  sixth  element  addressing  the  program  management  tasks 
spanning  the  five  phases8  (see  figure  1). 


8IDEALSM:  A  User’s  Guide  for  Software  Process  Improvement  (CMU/SEI-96-HB-001). 
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Figure  1 :  Simplified  Diagram  of  the  IDEALSM  Model 


•  Initiating:  During  this  phase,  an  organization  establishes  the 

management  structure  of  the  process  improvement  program,  defines 
and  assigns  roles  and  responsibilities,  allocates  initial  resources, 
develops  a  plan  to  guide  the  organization  through  the  first  three  phases 
of  the  program,  and  obtains  management  approval  and  funding.  Two  key 
organizational  components  of  the  program  management  structure 
established  during  this  phase  are  a  management  steering  group  and  a 
software  engineering  process  group  (SEPG).  Responsibility  for  this 
phase  rests  with  senior  management. 
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•  Diagnosing:  During  this  phase,  the  SEPG  appraises  the  current  level  of 
process  maturity9  to  establish  a  baseline  capability  against  which  to 
measure  progress  and  identifies  any  existing  process  improvement 
initiatives.  The  SEPG  then  uses  the  baseline  to  identify  weaknesses  and 
target  process  improvement  activities.  It  also  compares  these  targeted 
activities  with  any  ongoing  process  improvement  activities  and 
reconciles  any  differences.  Responsibility  for  this  phase  rests  primarily 
with  line  managers  and  practitioners. 

•  Establishing:  During  this  phase,  the  SEPG  prioritizes  the  process 
improvement  activities  and  develops  strategies  for  pursuing  them.  It 
then  develops  a  process  improvement  action  plan  that  details  the 
activities  and  strategies  and  includes  measurable  goals  for  the  activities 
and  metrics  for  monitoring  progress  against  goals.  Also  during  this 
phase,  the  resources  needed  to  implement  the  plan  are  committed  and 
training  is  provided  for  technical  working  groups,  who  will  be 
responsible  for  developing  and  testing  new  or  improved  processes. 
Responsibility  for  this  phase  resides  primarily  with  line  managers  and 
practitioners. 

•  Acting:  During  this  phase,  the  technical  working  groups,  formed  under 
the  establishing  phase,  create  and  evaluate  new  and  improved 
processes.  Evaluation  of  the  processes  is  based  on  pilot  tests  that  are 
formally  planned  and  executed.  If  the  tests  are  successful,  the  working 
groups  develop  plans  for  organization-wide  adoption  and 
institutionalization,  and  once  approved,  execute  them.  Responsibility 
for  this  phase  resides  primarily  with  line  managers  and  practitioners. 

•  Leveraging:  During  this  phase,  results  and  lessons  learned  from  earlier 
phases  are  assessed  and  applied,  as  appropriate,  to  enhance  the 
structures  and  plans  of  process  improvement  programs.  Responsibility 
for  this  phase  rests  primarily  with  senior  management. 

The  model’s  sixth  element,  continuous  program  management,  specifies 
management  structures  and  tasks  for  planning,  organizing,  directing, 
staffing,  and  monitoring  the  program.  Responsibility  for  this  element  rests 
with  senior  management. 


9SEI  has  developed  process  maturity  models  for  software  development,  software 
acquisition,  and  systems  engineering,  as  well  as  an  integrated  model  for  improving  software 
development,  acquisition,  and  maintenance.  (See  appendix  III  for  information  on  these 
models.) 
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Each  phase  of  the  IDEALSM  model  contains  several  recommended  tasks. 
Appendix  I,  which  describes  our  objectives,  scope,  and  methodology, 
identifies  all  tasks  for  each  phase. 


Results  in  Brief  The  DOD  components  that  we  reviewed  vary  in  how  they  compare  to  SEI’s 

IDEAL™  model.  In  particular,  the  Air  Force,  Army,  and  DFAS  generally 
satisfied  the  model’s  recommended  tasks,  as  did  certain  Navy  units. 
However,  DLA,  the  Marine  Corps,  and  other  Navy  units  did  not. 
Specifically,  DLA  does  not  have  an  SPI  program,  although  during  the  course 
of  our  review  the  DLA  Chief  Information  Officer  stated  that  she  intends  to 
establish  one.  Further,  although  the  Marine  Corps  is  performing  many  SPI 
activities,  core  tasks  associated  with  an  effective  SPI  program,  such  as  a 
plan  of  action  or  dedicated  resources  to  implement  recommended 
improvements,  are  missing.  Finally,  certain  Navy  units  also  do  not  have  SPI 
programs  aligned  with  the  IDEAL™  model,  although  one  is  performing  a 
few  of  the  model’s  recommended  tasks. 

The  four  components  with  SPI  programs  (Army,  Air  Force,  DFAS,  and  parts 
of  the  Navy)  are  using  different  management  strategies  for  directing  and 
controlling  their  respective  programs.  Nonetheless,  all  components  with 
SPI  programs  report  that  they  have  realized  benefits  in  product  quality  and 
productivity.  For  example,  DFAS  uses  a  centralized  management  approach 
and  reports  that  its  SPI  program  has  helped  decrease  development  costs  to 
about  one-third  lower  than  those  of  similar  organizations.  In  contrast,  the 
Army  uses  a  decentralized  approach  and  also  reports  that  the  SPI  program 
for  one  of  its  organizational  units  has  helped  it  almost  double  its 
productivity  in  developing  software. 

DOD-wide  activities  to  promote  and  leverage  component  SPI  programs  do 
not  exist.  According  to  the  IDEAL™  model,  leveraging  SPI  experiences  is 
fundamental  to  continuous  process  improvement.  While  two  organizational 
units  within  the  Office  of  the  Secretary  of  Defense  (OSD)  that  have 
important  leadership  roles  to  play  in  department  software  and  system 
processes  are  both  taking  steps  aimed  at  strengthening  DOD  software, 
these  steps  do  not  specifically  include  SPI.  In  particular,  OSD  does  not  have 
initiatives  under  way  or  planned  to  determine  where  in  DOD  SPI  programs 
do  and  do  not  exist  so  that  steps  can  be  taken  to  promote  programs  in 
component  units  where  they  do  not,  such  as  at  DLA.  Similarly,  actions  do 
not  exist  to  share  information  across  the  department  about  the  experiences 
of  successful  SPI  programs,  such  as  those  within  the  Army,  Navy,  Air  Force, 
and  DFAS.  According  to  OSD  officials,  uncertainty  about  the  costs  versus 
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benefits  of  SPI,  resource  constraints,  and  other  priorities  have  precluded 
such  a  focus.  Without  such  actions,  DOD  is  missing  opportunities  to  realize 
potential  SPI  benefits  in  all  DOD  components.  To  address  this,  we  are 
making  recommendations  to  the  Secretary  of  Defense. 

DOD  provided  written  comments  on  a  draft  of  this  report.  In  commenting, 
DOD  agreed  that  SPI  practices  should  be  used  and  encouraged  and  that 
information  about  best  practices  should  be  shared.  However,  DOD  stated 
that  it  is  premature  at  this  point  to  mandate  SPI  programs  throughout  the 
department,  as  we  recommend,  and  that  it  has  established  a  working  group 
to  review  how  best  to  proceed.  While  we  believe  that  sufficient  bases 
currently  exist  to  mandate  SPI,  particularly  in  light  of  the  evidence  in  this 
report  on  (1)  components  that  are  not  implementing  SPI  in  the  absence  of  a 
mandate  and  (2)  the  benefits  being  reported  by  components  that  are 
implementing  SPI,  we  do  not  view  DOD’s  desire  to  await  the  results  of  its 
working  group  as  being  unreasonable  or  inconsistent  with  our 
recommendations. 


Components’  SPI 
Program  Alignment 
With  SEI IDEALSM 
Model  Varies 


The  Army  and  Air  Force  units  that  we  reviewed,  as  well  as  DFAS  and  two  of 
the  four  Navy  units,  have  long-standing  SPI  programs  that  satisfy  almost 
every  task  recommended  in  the  IDEALSM  model  (see  table  1  for  a  summary 
of  how  each  component  and  its  units,  if  applicable,  compared  to  the 
model).  For  example,  in  1996  the  Secretary  of  the  Army  mandated  that  all 
software  development,  acquisition,  and  maintenance  activities  establish 
SPI  programs.  Further,  the  Army  requires  that  its  software  activities 
continually  improve  their  process  maturity  and  has  set  maturity  goals  for 
all  of  its  units.  Army  regulations  also  mandate  that  contractors  be 
evaluated  for  software  process  maturity.  Moreover,  the  two  specific  units 
within  the  Army  that  we  reviewed  have  SPI  management  structures,  plans, 
and  dedicated  resources.  In  addition,  these  units  have  continuously 
evolved  in  software  and  system  process  maturity  through  many  years  of 
assessing  their  baseline  process  capabilities,  implementing  new  and 
improved  process  initiatives,  reassessing  process  maturity,  and 
implementing  lessons  learned.  Both  Army  units  satisfy  all  IDEALSM  tasks. 
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Table  1 :  Comparison  of  Components  With  IDEALSM  Model 


Command/major  organizational 

Component 

unit 

Software/systems  unit 

Generally  satisfied? 

Army 

Communications-Electronics 

Software  Engineering  Center,  Fort  Monmouth, 

Yes 

Command 

NJ 

Aviation  and  Missile  Command 

Software  Engineering  Directorate,  Redstone 
Arsenal,  AL 

Yes 

Navy 

Naval  Aviation  Systems  Command, 
Patuxent  River,  MD 

Not  applicable3 

Yes 

Space  and  Naval  Warfare  Systems 

Systems  Center 

Command 

San  Diego,  CA 

Yes 

Chesapeake,  VA 

No 

Charleston,  SC 

No 

Air  Force 

Electronic  Systems  Center 

Standard  Systems  Group,  Maxwell  Air  Force 
Base,  AL 

Yes 

Materiel  Systems  Group,  Wright-Patterson  Air 
Force  Base,  OH 

Yes 

Air  Force  Academy,  Colorado 
Springs,  CO 

Not  applicable 

Yes 

Marine  Corps 

Marine  Corps  Systems  Command 

Marine  Corps  Tactical  Systems  Support 
Activity,  Camp  Pendleton,  CA 

No 

DFAS 

Information  and  Technology 
Directorate,  Arlington,  VA 

Not  applicable 

Yes 

DLA 

Headquarters,  Fort  Belvoir,  VA 

Not  applicable 

No 

aNot  applicable  indicates  that  SPI  responsibility  resides  with  the  “Command/major  organizational  unit” 
and  not  with  a  “Software/systems  unit.” 


In  contrast,  DLA,  the  Marine  Corps,  and  two  of  the  Navy’s  four  units  that 
we  reviewed  do  not  perform  important  IDEALSM  model  tasks.  In  particular, 
DLA  currently  does  not  satisfy  any  of  the  model’s  recommended  tasks. 
According  to  DLA  officials,  it  had  an  SPI  program  prior  to  1998,  but  at  that 
time  the  program  was  terminated  to  reduce  costs.  During  our  review,  DLA’s 
CIO  stated  that  the  agency  plans  to  begin  a  new  SPI  program  and  has  taken 
a  first  step  by  assigning  organizational  responsibility. 

The  Marine  Corps  has  many  SPI  activities  under  way  that  could  form  the 
foundation  of  a  program.  However,  it  is  not  performing  several  key  SPI 
tasks  that  are  fundamental  to  SPI  program  success.  For  example,  the 
Marine  Corps  has  assigned  responsibility  for  process  improvement,  and  it 
has  begun  assessing  its  software  process  maturity  to  establish  baseline 
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capability.  However,  it  is  not  using  this  baseline  as  a  basis  for  implementing 
recommended  improvements,  nor  does  it  have  an  SPI  plan  or  dedicated 
resources  for  these  activities.  As  such,  the  likelihood  of  the  Marine  Corps’ 
process  improvement  initiatives  producing  desired  results  is  diminished. 

Two  of  the  four  Navy  software/systems  units  that  we  reviewed  also  do  not 
have  SPI  programs  that  are  aligned  with  the  IDEALSM  model.  To  their  credit, 
however,  one  has  recently  taken  the  first  step  toward  initiating  a  program 
and  the  other  has  activities  under  way  that  could  form  the  beginnings  of  a 
program.  (See  appendix  IV  for  more  detailed  results  on  each  of  the 
components  that  we  reviewed.) 


Components’  SPI 
Management 
Approaches  Vary,  Yet 
All  Report  Positive 
Program  Results 


The  four  components  that  have  SPI  programs — Army,  Air  Force,  DFAS,  and 
parts  of  the  Navy — have  different  approaches  for  directing  and  controlling 
their  respective  programs,  ranging  from  centralized  to  highly  decentralized; 
each,  however,  reports  positive  results.  For  example,  DFAS  has  a 
centralized  approach,  with  its  headquarters  office  directing  and  controlling 
all  SPI  activities.  In  contrast,  the  Army,  Air  Force,  and  Navy  have 
decentralized  approaches  to  SPI  program  management.  The  Army,  which 
began  its  SPI  program  centrally,  has  since  delegated  SPI  responsibility  to  its 
commands,  which — in  the  case  of  the  two  commands  we  reviewed — have 
further  delegated  SPI  program  management  to  their  respective 
software/systems  units.  Similarly,  the  Air  Force  units  that  we  reviewed 
further  delegated  SPI  management  to  their  respective  software/systems 
units.  The  Navy  commands  follow  different  approaches — one  manages  its 
program  centrally  and  the  other  has  delegated  SPI  management  to  its 
software/systems  units. 


Despite  different  approaches,  each  DOD  component/unit  with  an  SPI 
program  reports  positive  effects  on  software/systems  quality.  DFAS,  for 
example,  reports  that  its  SPI  program  has  reduced  its  cost  to  deliver 
software  to  about  one-third  less  than  organizations  of  similar  size.  One 
Navy  software  activity  reports  reduced  costs,  improved  product  quality, 
and  a  7:1  return  on  its  SPI  investment.  An  Army  activity  reports  that  it  has 
almost  doubled  its  productivity  in  writing  software  for  new  systems 
because  of  improvements  made  under  its  SPI  program.  (See  appendix  IV 
for  more  detailed  information  on  the  approaches  and  reported  benefits  of 
the  components  that  we  reviewed.) 
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DOD-Wide  Efforts  to 
Promote  and  Leverage 
SPI  Programs  Do  Not 
Exist 


Within  OSD,  the  Assistant  Secretary  for  Command,  Control, 
Communications,  and  Intelligence  is  responsible  for  establishing  and 
implementing  DOD’s  policies,  processes,  programs,  and  standards 
governing  the  development,  acquisition,  and  operation  of  nonweapons 
systems  software  and  information  systems.10  Similarly,  the  Under  Secretary 
for  Acquisition,  Technology,  and  Logistics  is  responsible  for  establishing 
DOD  acquisition  policies  and  procedures.11  Accordingly,  OSD  has  an 
important  leadership  role  to  play  in  ensuring  that  DOD  components  reap 
the  maximum  possible  benefits  of  effective  SPI  programs.  Such  leadership 
can  include  dissemination  of  policies  and  guidance  promoting  SPI 
programs  and  activities,  knowledge  of  the  nature  and  extent  of 
components’  SPI  programs  and  activities,  associated  lessons  learned  and 
best  practices,  and  facilitation  of  SPI  knowledge-sharing  across  DOD 
components. 


Both  OSD  organizational  units  have  efforts  under  way  aimed  at  improving 
some  aspects  of  DOD’s  ability  to  develop  and  acquire  software  and 
systems.  For  example,  they  have  established  teams  to  conduct  software 
acquisition  maturity  assessments  and  established  a  software  collaborators 
group.  They  also  are  collecting  software  metrics  and  establishing  training 
for  managers. 


However,  OSD  has  no  SPI  actions  under  way  or  planned,  such  as  issuing 
policy  and  guidance  on  SPI  programs;  determining  where  in  DOD  SPI 
programs  do  and  do  not  exist;  promoting  the  establishment  of  programs  in 
component  units,  such  as  DLA,  where  they  do  not  exist;  and  sharing 
knowledge  across  DOD  about  the  experiences  of  reportedly  successful  SPI 
programs,  such  as  those  within  the  Army,  Air  Force,  DFAS,  and  parts  of  the 
Navy.  According  to  OSD  officials,  uncertainty  about  the  costs  versus 
benefits  of  SPI,  resource  constraints,  and  other  priorities  have  precluded 
such  a  focus.  However,  as  stated  earlier  in  this  report,  various 
organizations,  including  some  DOD  components,  report  positive  returns  on 
investments  from  SPI  programs  that  argue  for  SPI  being  treated  as  a 
funding  priority. 


10DOD  Directive  5137.1. 
nDOD  Directive  5134.1. 
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Conclusions  Several  DOD  components  have  SPI  programs  that  are  aligned  closely  to  the 

best  practices  embodied  in  the  SEI IDEALSM  model  and  thus  provide 
excellent  examples  of  SPI.  However,  such  programs  are  lacking  in  other 
parts  of  the  department.  Where  they  exist,  these  programs  are  being 
credited  with  producing  higher  quality  software  and  systems  products 
faster  and  at  less  expense,  whether  managed  in  a  centralized  or 
decentralized  fashion. 

OSD  has  an  important  leadership  role  to  play  in  expanding  SPI  across  the 
department.  In  particular,  it  can  seize  opportunities  to  build  upon  and 
leverage  the  existing  base  of  SPI  programs  within  DOD’s  components  and 
help  ensure  that  all  of  its  components  realize  the  strategic  value  (i.e., 
benefits  that  exceed  costs)  that  both  private  and  public-sector 
organizations,  including  some  DOD  components,  attribute  to  these 
programs.  While  OSD  is  faced  with  making  funding  choices  among 
competing  leadership  initiatives,  such  as  its  efforts  to  conduct  software 
acquisition  maturity  assessments  and  collect  software  metrics,  these  are 
some  of  the  very  tasks  that  are  embedded  within  an  effective  SPI  program. 
Thus,  by  ensuring  that  DOD  components  have  effective  SPI  programs,  OSD 
can  leverage  programs  to  indirectly  accomplish  its  other  high-priority 
initiatives  as  well. 


Recommendations 
Executive  Action 


To  strengthen  DLA,  Marine  Corps,  and  Navy  software  and  systems 
development,  acquisition,  and  engineering  processes,  we  recommend  that 
the  Secretary  of  Defense  direct  the  Director  of  DLA,  the  Commandant  of 
the  Marine  Corps,  and  the  Secretary  of  the  Navy  to  establish  SPI  programs 
where  this  report  shows  none  currently  exist.  In  so  doing,  these  officials 
should  consider  following  the  best  practices  embodied  in  the  SEI  IDEALSM 
model  and  drawing  from  the  experiences  of  the  Army,  Air  Force,  DFAS,  and 
some  Navy  units. 


Further,  to  strengthen  DOD-wide  SPI,  we  recommend  that  the  Secretary  of 
Defense  direct  the  Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications,  and  Intelligence,  in  collaboration  with  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics,  to  (1)  issue 
a  policy  requiring  DOD  components  that  are  responsible  for 
systems/software  development,  acquisition,  or  engineering  to  implement 
SPI  programs,  and  (2)  develop  and  issue  SPI  guidance  and,  in  doing  so, 
consider  basing  this  guidance  on  the  SEI  IDEAL™  model  and  the  positive 
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examples  of  SPI  within  the  Army,  Air  Force,  DFAS,  and  some  Navy  units 
cited  in  this  report. 

We  also  recommend  that  the  Secretary  direct  the  Assistant  Secretary  for 
Command,  Control,  Communications,  and  Intelligence  to  (1)  annually 
determine  the  components’  compliance  with  the  SPI  policy  and 
(2)  establish  and  promote  a  means  for  sharing  SPI  lessons  learned  and  best 
practices  knowledge  throughout  DOD. 


Agency  Comments  and 
Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  the  Deputy  Assistant 
Secretary  of  Defense  for  Command,  Control,  Communications,  and 
Intelligence,  who  is  also  the  DOD  Deputy  Chief  Information  Officer  (CIO), 
agreed  with  the  report’s  message  that  SPI  practices  should  be  used  and 
encouraged,  and  that  information  about  SPI  practices  should  be  shared 
among  DOD  components.  To  this  end,  and  since  receiving  a  draft  of  this 
report,  the  Deputy  CIO  stated  that  the  Under  Secretary  of  Defense 
(Acquisition,  Technology,  and  Logistics)  has  established  a  working  group12 
that  is,  among  other  things,  to  develop  a  plan  for  implementing  SPI. 
According  to  the  Deputy  CIO,  this  plan  will  be  ready  for  internal  review  in 
April  2001. 


Further,  the  Deputy  CIO  stated  that  a  January  2001  revision  to  DOD 
Regulation  5000.2-R13  represents  a  policy  step  toward  addressing  software 
improvement  by  including  in  the  regulation  a  section  on  software 
management.  According  to  the  Deputy  CIO,  while  this  section  does  not 
specifically  call  for  an  SPI  program,  the  regulation  provides  guidance  for 
improving  software  by  using,  for  example,  SEI’s  Capability  Maturity  Model 
level  3  or  its  equivalent  for  major  acquisition  programs  with  procurement 
costs  in  excess  of  $2.19  billion.14 


12This  group  is  called  the  Independent  Expert  Program  Review  Working  Group.  It  was 
established  in  January  2001. 

13Interim  Regulation  5000.2-R,  “Mandatory  Procedures  for  Major  Defense  Acquisition 
Programs  and  Major  Automated  Information  System  Acquisition  Programs”  (January  4, 
2001). 

14Interim  Regulation  5000.2-R  refers  to  these  programs  as  Acquisition  Category  (ACAT)  1 
programs. 
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In  light  of  the  above,  the  Deputy  CIO  stated  that  DOD  agreed  with  our 
recommendation  to  establish  and  promote  a  means  for  sharing  SPI  lessons 
learned  and  best  practices  knowledge  throughout  DOD,  and  added  that  a 
DOD  steering  group,16  which  was  chartered  during  the  course  of  our 
review,  has  been  assigned  responsibility  for  this  function.  However,  the 
Deputy  CIO  disagreed  with  our  recommendation  that  DOD  issue  a  policy  to 
mandate  SPI  programs  for  ah  DOD  components  and  their  relevant 
activities.  According  to  the  Deputy  CIO,  establishing  a  policy  requiring  or 
otherwise  directing  DOD  components  that  do  not  have  SPI  programs  to 
implement  them  would  be  premature  at  this  time  because  there  are 
insufficient  data  to  justify  the  sole  use  of  the  SEI IDEALSM  model  and  that 
unless  a  specific  model  were  used,  compliance  with  such  a  policy  or 
directive  would  be  problematic.  Therefore,  the  Deputy  CIO  stated  a 
decision  regarding  the  issuance  of  DOD-wide  policy  mandating  the 
implementation  of  SPI  programs  would  not  be  made  until  the  work  group 
reports  its  results  and  develops  its  plan  for  implementing  SPI.  At  this  point 
and  without  the  work  group’s  findings,  according  to  the  Deputy  CIO, 
issuance  of  SPI  guidance  (as  opposed  to  “policy”)  would  be  “a  more 
beneficial  approach.” 

In  our  view,  the  Deputy  CIO’s  comments  are  not  inconsistent  with  our 
recommendations,  and  our  point  of  disagreement  appears  to  center  around 
simply  the  timing  of  actions  rather  than  the  recommended  actions 
themselves.  Specifically,  while  we  continue  to  believe  that  sufficient  bases 
currently  exist  for  issuance  of  a  DOD  SPI  policy  requirement,  especially  in 
light  of  the  evidence  in  our  report  that  (1)  without  this  requirement  not  ah 
components  are  implementing  SPI  and  (2)  those  components  that  are 
currently  implementing  SPI  are  reporting  substantial  benefits,  it  is 
reasonable  for  DOD  to  await  its  work  group’s  results  before  making  a 
decision  on  how  to  proceed.  Further,  we  agree  with  the  Deputy  CIO’s 
comment  that  there  are  insufficient  data  to  justify  citing  in  DOD  policy  the 
SEI  IDEALSM  model  as  the  single  model  for  SPI.  Our  report  recognizes  that 
not  all  of  the  DOD  components  that  we  cited  as  having  effective  SPI 
programs  are  using  the  same  model.  As  a  result,  our  recommendations  did 
not  prescribe  a  specific  SPI  model.  Instead,  we  recommended  that  in 
developing  SPI  policy  and  associated  guidance,  DOD  should  consider 
basing  this  guidance  on  the  SEI  IDEAL™  model  as  well  as  the  positive 


I5This  group  is  called  the  Software  Intensive  Systems  Steering  Group.  It  was  chartered  in 
September  2000. 
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examples  of  SPI  within  the  Army,  Air  Force,  DFAS,  and  some  Navy  units 
cited  in  the  report. 

Regarding  the  Deputy  CIO’s  comment  that  DOD  has  recently  revised  DOD 
Regulation  5000.2-R  to  include  guidance  for  improving  software 
management  through  the  use  of,  for  example,  SEI’s  Capability  Maturity 
Model  level  3,  we  note  that  level  3  requirements  include  performance  of 
process  improvement  practices  that  are  expanded  upon  by  the  SEI 
IDEALSM  model.  Additionally,  we  note  that  the  regulation  does  not  apply  to 
all  DOD  software/system  programs  but,  rather,  only  to  acquisition 
programs  that  exceed  a  certain  dollar  threshold.  Therefore,  the  revised 
regulation  does  not  fulfill  the  intent  of  our  recommendations. 

DOD’s  written  comments,  along  with  our  responses,  are  reproduced  in 
appendix  II. 


We  are  sending  copies  of  this  report  to  Senator  John  Warner,  Senator  Carl 
Levin,  Senator  Ted  Stevens,  Senator  Daniel  Inouye,  and  to  Representative 
Bob  Stump,  Representative  Ike  Skelton,  and  Representative  C.W.  Bill 
Young,  in  their  capacities  as  Chairmen,  Ranking  Members,  or  Ranking 
Minority  Members  of  Senate  and  House  Committees  and  Subcommittees. 
In  addition,  we  are  sending  copies  of  this  report  to  the  Secretaries  of  the 
Army,  Navy,  and  Air  Force;  the  Commandant  of  the  Marine  Corps;  the 
Directors  of  DLA  and  DFAS;  and  the  Director,  Office  of  Management  and 
Budget.  Copies  will  also  be  available  at  GAO’s  web  site,  www.gao.gov. 

If  you  have  any  questions  about  this  report,  please  contact  me  at  (202)  512- 
3439  or  by  e-mail  at  hiter@gao.gov.  Key  contributors  to  this  report  are  listed 
in  appendix  V. 


Randolph  C.  Hite 

Director,  Information  Technology  Systems  Issues 
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Appendix  I 


Objectives,  Scope,  and  Methodology 


Our  objectives  were  to  (1)  compare  selected  DOD  components’  SPI 
programs  against  SEI’s  IDEALSM  model,  which  is  a  recognized  best 
practices  model;  (2)  determine  how  these  components  have  approached 
management  of  their  SPI  programs  and  what  program  results  they  are 
reporting;  and  (3)  determine  what  DOD-wide  efforts  are  under  way  to 
promote  and  leverage  the  components’  SPI  programs.  The  selected 
components  include  all  four  services — Army,  Air  Force,  Navy,  Marine 
Corps — and  two  DOD  agencies  that  have  large,  software-intensive  system 
modernization  programs  under  way — the  Defense  Finance  and  Accounting 
Service  (DFAS)  and  the  Defense  Logistics  Agency  (DLA).1 

To  address  the  first  objective,  we  reviewed  the  components’  respective 
information  technology  strategic  plans  as  well  as  available  SPI  policies, 
guidance,  and  program  documentation,  and  interviewed  headquarters 
officials  from  each  component.  Using  this  information,  we  first  ascertained 
whether  SPI  programs  or  activities  existed  for  a  component,  and  if  so,  how 
they  were  organized  and  structured.  For  the  components  in  which  we 
found  SPI  programs  or  activities,  we  then  identified  the  units  within  the 
components  responsible  for  implementing  those  programs  and  activities. 
In  instances  in  which  these  responsibilities  were  decentralized  (Army,  Air 
Force,  and  Navy),  we  worked  with  component  headquarters  and  command 
officials  to  select  at  least  two  units  in  each  component  that  collectively 
(1)  had  missions  involving  both  software-intensive  weapons  and  business 
systems  and  (2)  were  responsible  for  the  largest  percentages  of  software 
and  systems  development,  acquisition,  and  engineering  activities  within 
each  component.  Table  2  shows  the  DOD  components  and 
software/systems  units  where  we  reviewed  SPI  programs  and  activities. 
Where  “not  applicable”  is  indicated  in  the  table,  SPI  responsibility  resided 
at  the  “Command/major  organizational  unit,”  and  therefore  our  work  did 
not  extend  to  a  “Software/systems  unit.” 


'DFAS  plans  to  spend  over  $2.2  billion  by  2007  to  modernize  its  finance  and  accounting 
systems.  DLA  plans  to  spend  about  $525  million  by  2005  to  modernize  its  business  systems. 
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Appendix  I 

Objectives,  Scope,  and  Methodology 


Table  2:  Software/Systems  Units  Selected  for  Review 


Component 

Command/major  organizational  unit 

Software/systems  unit 

Army 

Communications-Electronics  Command 

Software  Engineering 

Center 

Aviation  and  Missile  Command 

Software  Engineering 
Directorate 

Navy 

Naval  Aviation  Systems  Command 

Not  applicable 

Space  and  Naval  Warfare  Systems 
Command 

Systems  Center 

San  Diego 

Chesapeake 

Charleston 

Air  Force 

Electronic  Systems  Center 

Standard  Systems  Group 
Materiel  Systems  Group 

Air  Force  Academy 

Not  applicable 

Marine  Corps 

Marine  Corps  Systems  Command 

Marine  Corps  Tactical 
Systems  Support  Activity 

DFAS 

Information  and  Technology  Directorate 

Not  applicable 

DLA 

Fleadquarters 

Not  applicable 

For  each  unit  that  we  identified  as  being  responsible  for  implementing  an 
SPI  program  or  activities,  we  analyzed  relevant  SPI  program 
documentation,  including  program  descriptions,  plans,  budgets,  and 
progress  and  performance  measures  and  reports,  and  interviewed  program 
officials.  We  then  compared  this  information  with  the  SPI  tasks  specified 
and  described  in  SEI’s  IDEALSM  model  to  determine  whether  the  program 
satisfied  the  model. 

Designed  to  assist  organizations  in  implementing  and  managing  effective 
SPI  programs,  the  SEI-developed  IDEALSM  model  comprises  five  specific 
phases;  a  sixth  element  addresses  overall  management  of  the  five  phases. 
Table  3  provides  more  information  about  the  tasks  involved  in  each  phase. 
Table  4  lists  every  task  included  under  each  phase. 
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Appendix  I 

Objectives,  Scope,  and  Methodology 


Table  3:  Phases  of  the  IDEALSM  Model 

Phase 

Description  of  typical  tasks 

Initiating 
(10  tasks) 

Senior  managers  establish  SPI  program  structure,  define  roles,  allocate  resources,  and  develop  a  plan  to  guide 
the  organization  through  the  Establishing  phase;  management  commitment  and  funding  are  obtained.  Two  key 
structural  components  established  in  this  phase  are  a  management  steering  group  and  a  software  engineering 
process  group  (SEPG). 

Diagnosing 
(6  tasks) 

SEPG — with  line  managers  and  practitioners — appraises  the  level  of  software  process  maturity  to  obtain  a 
baseline  capability  against  which  to  measure  progress.  Any  existing  process  improvement  initiatives  are 
identified,  along  with  weaknesses  and  needed  improvement  activities. 

Establishing 
(14  tasks) 

SEPG,  line  managers,  and  practitioners  prioritize  SPI  activities  and  develop  strategies  and  an  action  plan, 
including  measurable  goals  and  metrics  for  monitoring  progress.  Resources  needed  to  implement  the  plan  are 
committed,  and  training  is  provided  for  technical  working  groups  that  will  develop  and  test  new  or  improved 
processes. 

Acting 
(10  tasks) 

Pilot  tests  are  planned  and  executed  to  evaluate  new  and  improved  processes  created  by  the  technical  working 
groups.  If  tests  succeed,  plans  are  developed  for  organizationwide  adoption,  institutionalization,  and  execution. 

Leveraging 
(7  tasks) 

Senior  managers  assess  and  apply  lessons  learned  to  enhance  the  SPI  program’s  structure  and  plans. 

Managing 
(6  tasks) 

Senior  managers  ensure  that  decisions  made  are  based  on  organizational  needs  and  that  the  management 
structure  guides  and  prioritizes  SPI  tasks. 

Table  4:  Phases  and  Tasks  of  the  IDEALSM  Model 


Phase  Task 

Initiating  Organize  discovery  team  to  develop  a  proposal  to  management  for 
launching  SPI  program 

Identify  business  needs  and  drivers  for  improvement 
Build  an  SPI  proposal 
Educate  and  build  support 

Obtain  approval  for  SPI  proposal  and  initial  resources 

Establish  SPI  infrastructure 

Assess  the  climate  for  SPI 

Define  general  SPI  goals 

Define  guiding  principles  of  SPI  program 

Launch  the  program 
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(Continued  From  Previous  Page) 

Phase  Task 

Diagnosing  Determine  what  baseline(s)  are  needed 

Plan  for  baseline(s) 

Conduct  baseline(s) 

Present  findings 

Develop  final  findings  and  recommendations  report 
Communicate  findings  and  recommendations  to  organization 
Establishing  Select  and  get  training  in  a  strategic  planning  process 
Review  organization’s  vision 
Review  organization’s  business  plan 
Determine  key  business  issues 
Review  past  improvement  efforts 
Describe  motivations  to  improve 

Identify  current  and  future  (planned)  improvement  efforts 
Finalize  roles  and  responsibilities  of  infrastructure  entities 
Prioritize  activities  and  develop  improvement  agenda 

Reconcile  existing  planned  improvement  efforts  with  baseline  findings  and 
recommendations 

Transform  general  SPI  goals  to  measurable  goals 
Create/update  SPI  strategic  plan 

Build  consensus,  review,  approve  SPI  strategic  plan  and  commit 
resources 

Form  technical  working  group 

Acting  Complete  tactical  plan  for  technical  working  group 

Develop  solutions 
Pilot  potential  solutions 
Select  solution  providers 
Determine  long-term  support  needs 
Develop  rollout  strategy  and  plan  template 
Package  improvement  and  turn  over  to  SEPG 
Disband  technical  working  group 
Roll  out  solution 
Transition  to  long-term  support 
Leveraging  Gather  lessons  learned 
Analyze  lessons  learned 
Revise  organizational  approach 
Review  sponsorship  and  commitment 
Establish  high-level  goals 
Develop  new/revised  SPI  proposal 
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(Continued  From  Previous  Page) 


Phase 

Task 

Continue  with  SPI 

Managing 

Set  the  stage  for  SPI 

Organize  the  SPI  program 

Plan  the  SPI  program 

Staff  the  SPI  program 

Monitor  the  SPI  program 

Direct  the  SPI  Program 

To  address  the  second  objective,  we  analyzed  the  aforementioned 
information,  conducted  additional  interviews,  and  reviewed  additional 
program  information  from  the  component  units  to  which  SPI  management 
responsibility  had  been  delegated.  As  part  of  this  objective,  we  also 
reviewed  program  progress  and  performance  reports  and  discussed 
program  accomplishments  with  responsible  officials  to  identify  examples 
of  SPI  benefits.  We  then  analyzed  each  component’s  SPI  program  results  in 
relation  to  its  program  management  approach  to  determine  whether  any 
patterns  were  evident.  We  did  not  independently  validate  components’ 
reported  accomplishments  and  benefits. 

To  address  the  third  objective,  we  interviewed  responsible  component 
officials,  reviewed  supporting  records  and  documentation,  and  visited 
Internet  sites  to  identify  SPI  program  best  practices  and  lessons  learned, 
along  with  what  efforts  are  being  made  to  share  these  with  other  activities 
and  components  throughout  the  department.  We  also  identified  two  offices 
within  the  Office  of  the  Secretary  of  Defense  (OSD)  that  have  responsibility 
and  activities  underway  relating  to  the  advancement  of  software  and 
system  management  practices  in  the  department— the  Office  of  the  Deputy 
Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics;  and 
the  Office  of  the  Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications,  and  Intelligence.  For  each  office,  we  analyzed 
documentation  describing  their  respective  ongoing  and  planned  activities 
and  interviewed  officials.  In  doing  so,  we  focused  on  identifying  any 
activities  that  specifically  promoted  and  leveraged  SPI  programs  and 
activities  under  way  throughout  DOD.  We  also  discussed  with  SPI  program 
officials  in  each  component  their  awareness  of  the  OSD  efforts. 

We  performed  our  work  at  Army  headquarters,  the  Pentagon,  Arlington, 
Virginia;  and  interviewed  officials  and  reviewed  documentation  from  the 
Communications-Electronics  Command  Software  Engineering  Center  at 
Fort  Monmouth,  New  Jersey;  and  the  Aviation  and  Missile  Command 
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Software  Engineering  Directorate  at  Redstone  Arsenal,  Alabama.  We  also 
performed  our  work  at  Navy  headquarters  in  Arlington,  Virginia;  and 
interviewed  officials  and  reviewed  documentation  from  the  Naval  Aviation 
Systems  Command  at  Patuxent  River,  Maryland;  and  the  Space  and  Naval 
Warfare  Systems  Command  Centers  at  San  Diego,  California;  Chesapeake, 
Virginia;  and  Charleston,  South  Carolina.  We  also  interviewed  officials  and 
reviewed  documentation  from  the  Air  Force’s  Electronic  Systems  Center 
Standard  Systems  Group  at  Maxwell  Air  Force  Base,  Alabama;  the  Materiel 
Systems  Group  at  Wright-Patterson  Air  Force  Base,  Ohio;  and  the  Air  Force 
Academy  in  Colorado  Springs,  Colorado.  We  also  performed  our  work  at 
Marine  Corps  headquarters  in  Arlington,  Virginia;  and  interviewed  officials 
and  reviewed  documentation  from  the  Marine  Corps  Systems  Command  in 
Quantico,  Virginia;  and  the  Marine  Corps  Tactical  Systems  Support  Activity 
at  Camp  Pendleton,  California.  We  also  performed  work  at  DFAS 
headquarters  in  Arlington,  Virginia;  and  DLA  headquarters  at  Fort  Belvoir, 
Virginia.  We  conducted  our  work  from  March  through  December  2000,  in 
accordance  with  generally  accepted  government  auditing  standards. 
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Note:  GAO  comments 
supplementing  those  in  the 
report  text  appear  at  the 
end  of  this  appendix. 


OFFICE  OF  THE  ASSISTANT  SECRETARY  OF  DEFENSE 
6000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-6000 

March  2,  2001 


COMMAND,  CONTROL, 

COMMUNICATIONS,  AND 
INTELLIGENCE 

Mr.  Joel  C.  Willemssen 
Managing  Director 
Information  Technology  Issues 
U.S.  General  Accounting  Office 
Washington,  D.C.  20548 

Dear  Mr.  Willemssen: 

Thank  you  for  the  opportunity  to  review  the  GAO  Draft  Report,  “DoD  INFORMATION 
TECHNOLOGY :  Software  and  System  Process  Improvement  Programs  Vary  in  Use  of  Best 
Practices,”  January  4,  2001  (GAO  code  51 1686/OSD  Case  3026).  After  careful  review,  we 
partially  concur  with  the  recommendations  presented  in  the  report. 

We  agree  that  software  process  improvement  (SPI)  practices  should  be  used,  encouraged, 
and  information  shared.  However,  it  is  premature  to  issue  a  DoD- wide  policy  mandating  the 
implementation  of  a  SPI  program  until  the  results  are  received  from  the  Independent  Expert 
Program  Review  Working  Group  (IEPRWG). 

The  IEPR  WG,  established  by  the  Under  Secretary  of  Defense  (Acquisition,  Technology, 
and  Logistics),  will  develop  an  implementation  plan  that  will  directly  address  software  process 
improvement.  This  plan  will  be  ready  for  internal  review  in  April  2001. 

The  Department  has  taken  other  steps  in  policy  to  address  software  improvements.  DoD 
Regulation  5000.2-R  has  been  revised  to  include  a  section  on  software  management.  Although 
this  section  does  not  specify  a  software  process  improvement  program,  it  does  provide  guidance 
for  improving  software  such  as  the  implementation  of  the  Software  Engineering  Institute’s 
Capability  Maturity  Model  Level  3,  or  its  equivalent  for  ACAT  1  programs. 

Specific  comments  on  the  Report  and  its  recommendations  are  provided,  along  with  the 
software  management  additions  to  5000.2-R,  in  the  enclosure. 


Deputy  Assistant  Secretary  of  Defense 
Deputy  Chief  Information  Officer 

Enclosure 

6 
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GAO  DRAFT  REPORT  -  DATED  JANUARY  4,  2001 
GAO  CODE  511686/OSD  CASE  3026 


“DOD  INFORMATION  TECHNOLOGY :  Software  and  Systems  Process  Improvement 
Programs  Vary  in  Use  of  Best  Practices” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  RECOMMENDATIONS 


RECOMMENDATION  1:  The  GAO  recommended  that  the  Secretary  of  Defense  direct  the 
Director  of  DLA,  the  Commandant  of  the  Marine  Corps,  and  the  Secretary  of  the  Navy  to 
establish  SPI  programs  where  this  report  shows  none  currently  exist.  In  so  doing,  these  officials 
should  consider  following  the  best  practices  embodied  in  the  SEI  IDEAL™  model  and  drawing 
from  the  experience  of  the  Army,  Air  Force,  DFAS  and  some  Navy  units,  (p.  13/Draft  Report) 

See  comment  1.  DqD  RESPONSE:  Non-concur.  A  more  beneficial  approach  would  be  for  the  Assistant 

Secretary  of  Defense  for  Command,  Control,  Communications  and  Intelligence,  in  collaboration 
with  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  to  issue  DoD 
guidance  for  a  Software  Process  Improvement  best  practice  to  be  adopted  DoD-wide.  The 
issuance  of  a  formal  policy  would  create  a  problem  in  determining  compliance  unless  a  specific 
SPI  model  was  recommended.  Currently  there  is  insufficient  data  for  the  justification  of  use  of  a 
single  SPI  model  to  be  cited  in  formal  DoD-wide  policy. 

The  Independent  Expert  Program  Review  Working  Group  (IEPR  WG),  established  by  the 
Under  Secretary  of  Defense  (Acquisition,  Technology,  and  Logistics),  will  develop  an 
implementation  plan  that  will  influence  software  process  improvement.  This  plan  will  be  ready 
for  internal  review  in  April  2001 . 

OSD  is  developing  a  set  of  measures  to  indicate  causes  of  program  cost,  schedule  and 
performance  deficiencies  related  to  software  management.  These  measures  are  based  upon 
higher  level  goals  and  are  analogous  to  the  initial  phases  of  the  IDEAL™  model.  The  measures, 
once  implemented,  will  begin  to  assess  software  process  improvement  efforts. 

RECOMMENDATION  2:  The  GAO  recommended  that  the  Secretary  of  Defense  direct  the 
Assistant  Secretary  of  Defense  for  Command,  Control,  Communications  and  Intelligence,  in 
collaboration  with  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  to 
(1)  issue  a  policy  requiring  DoD  components  that  are  responsible  for  systems/software 
development,  acquisition,  or  engineering  to  implement  SPI  programs,  and  (2)  develop  and  issue 
SPI  guidance,  and  in  doing  so,  consider  basing  this  guidance  on  the  SEI  IDEAL™  model  and  the 
positive  examples  of  SPI  within  the  Army,  Air  Force,  DFAS,  and  some  Navy  units  cited  in  this 
report,  (p.  1 3/Draft  Report) 

See  comment  2.  DoD  RESPONSE:  Non-concur.  It  is  premature  to  issue  a  DoD-wide  policy  mandating  the 

implementation  of  a  SPI  program  until  the  results  are  received  from  the  IEPR  WG. 
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See  comment  3. 


RECOMMENDATION  3:  The  GAO  recommended  that  the  Secretary  direct  the  Assistant 
Secretary  of  Defense  for  Command,  Control,  Communications  and  Intelligence  to  (1)  annually 
determine  the  Components’  compliance  with  the  SPI  policy;  and  (2)  establish  and  promote  a 
means  for  sharing  SPI  lessons  learned  and  best  practices  knowledge  throughout  DoD.  (p. 
13/Draft  Report) 

DoD  RESPONSE:  Partially  concur.  The  Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications  and  Intelligence  should  not  annually  determine  the  components’  compliance 
with  the  SPI  policy  for  the  reason  stated  above.  There  would  be  significant  difficulty  in 
establishing  a  baseline  measure  to  determine  compliance  with  any  SPI  policy  or  practice  without 
requiring  the  implementation  of  a  specific  model. 

We  concur  that  SPI  lessons  learned  and  best  practices  should  be  promoted  throughout 
DoD  and  this  is  one  of  the  functions  of  the  Software  Intensive  Systems  Steering  Group  (SIS  SG) 
(charter  attached).  This  SIS  SG  contains  members  from  each  Service  and  C3I,  and  is  chaired  by 
the  Deputy  Under  Secretary  of  Defense  for  Science  and  Technology. 
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General  C3I  Comment 

Software  management  sections  have  been  added  to  DoD  Regulation  5000.2-R 
“Mandatory  Procedures  for  Major  Defense  Acquisition  Programs  (MDAPs)  and  Major 
Automated  Information  System  (MAIS)  Acquisition  Programs”  that  have  bearing  on  SPI.  This 
is  the  first  major  policy  statement  to  specifically  address  software  management  that  promotes  a 
DoD-wide  SPI  program.  Section  5.2.6,  et.  seq.  of  the  DoD  5000.2-R  Regulation  are  provided 
below: 

5.2.6  Software  Management 

The  PM  shall  manage  and  engineer  software-intensive  systems  using  best  processes 
and  practices  known  to  reduce  cost,  schedule,  and  performance  risks. 

5.2.6.1  General 

The  PM  shall  base  software  systems  design  and  development  on  systems  engineering 
principles,  to  include  the  following: 

•  Develop  architectural  based  software  systems  that  support  open  system 
concepts;  exploit  COTS  computer  systems  products;  and  allow  incremental 
improvements  based  on  modular,  reusable,  extensible  software; 

•  Identify  and  exploit,  where  practicable,  government  and  commercial  software 
reuse  opportunities  before  developing  new  software; 

•  Select  the  programming  language  in  context  of  the  systems  and  software 
engineering  factors  that  influence  overall  life-cycle  costs,  risks,  and  the 
potential  for  interoperability; 

•  Use  DoD  standard  data  and  follow  data  administrative  policies  in  DoDD 
8320.11, 

•  Select  contractors  with  domain  experience  in  developing  comparable  software 
systems;  with  successful  past  performance;  and  with  a  mature  software 
development  capability  and  process.  Contractors  performing  software 
development  or  upgrade(s)  for  use  in  an  ACAT  I  program  shall  undergo  an 
evaluation,  using  either  the  tools  developed  by  the  Software  Engineering 
Institute  (SEI),  or  those  approved  by  the  DoD  Components  and  the  Deputy 
Under  Secretary  of  Defense  (Science  and  Technology)  (DUSD(S&T)).  At  a 
minimum,  full  compliance  with  SEI  Capability  Maturity  Model  Level  3,  or  its 
equivalent  in  an  approved  evaluation  tool,  is  the  Department's  goal.  However, 
if  the  prospective  contractor  does  not  meet  full  compliance,  a  risk  mitigation 
plan  and  schedule  shall  be  prepared  to  describe,  in  detail,  actions  that  will  be 
taken  to  remove  deficiencies  uncovered  in  the  evaluation  process.  The  risk 
mitigation  plan  shall  require  PM  approval.  The  DUSD(S&T)  shall  define  Level 
3  equivalence  for  approved  evaluation  tools.  The  evaluation  shall  examine  the 
business  unit  proposed  to  perform  the  work.  Evaluation  results  shall  remain 
valid  for  a  period  of  2  years. 

•  Use  a  software  measurement  process  in  planning  and  tracking  the  software 
program,  and  to  assess  and  improve  the  software  development  process  and 
the  associated  software  product.  Provide  those  measures  to  the  appropriate 
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OSD  oversight  office.  For  example,  Major  Automated  information  System  PMs 
shall  follow  the  process  described  in  the  Practical  Software  and  System 
Measurement  Guidebook  (http://www.psmsc.com/). 

•  Assess  information  operations  risks  ( DoDD  S-3600.1")  using  techniques  such 
as  independent  expert  reviews; 

•  Prepare  for  life-cycle  software  support  or  maintenance  by  developing  or 
acquiring  the  necessary  documentation,  host  systems,  test  beds,  and 
computer-aided  software  engineering  tools  consistent  with  planned  support 
concepts;  and  by  planning  for  transition  of  fielded  software  to  the 
support/maintenance  activity; 

•  Track  COTS  software  purchases  and  maintenance  licenses;  and 

•  Structure  a  software  development  process  that  recognizes  that  emerging 
requirements  will  require  modification  to  software  over  the  life  cycle  of  the 
system.  In  order  to  deliver  truly  state-of-th e-software,  this  process  should  allow 
for  periodic  software  enhancements. 

5.2.6.2  Software  Spiral  Development 

When  acquiring  software  for  a  system,  the  PM  shall  plan  a  spiral  development  process 
for  both  evolutionary  and  single-step-to-full-capability  acquisition  strategies.  A  cyclical, 
iterative  build-test-fix-test-deploy  process  characterizes  spiral  development  and  yields 
continuous  improvements  in  software.  Each  software  release  draws  upon  the 
experience  and  lessons  of  previous  releases.  The  spiral  development  process  shall 
accomplish  the  following: 

•  Facilitate  requirements  changes  resulting  from  operational  mission  needs, 
technology  opportunities,  experimentation  results,  and  technology 
obsolescence. 

•  Incorporate  T&E  of  operational  effectiveness,  suitability,  and  supportability 
using  experimentation,  demonstration,  rigorous  testing,  or  certification. 

o  The  T&E  process  shall  be  continuous  throughout  the  system  life  cycle 
and  involve  the  user,  contractor,  program  office,  and  test  community, 
o  The  T&E  process  shall  consider  the  near  continuous  nature  of  change  in 
the  baseline  and  use  techniques  such  as  regression  testing  to  ensure 
that  existing  functionality  has  not  been  compromised, 
o  The  PM  shall  consider  the  risks  and  extent  of  change  impacts  to  enable 
a  cost-effective,  yet  rigorous  T&E  process. 

•  Implement  configuration,  change,  and  data  management. 

o  Documented  actual  deployed  capability  provides  the  starting  point  for 
development  of  the  next  improvement  release  and  provides  a  baseline 
for  verification,  training,  etc. 

o  The  PM  shall  implement  a  configuration  control  board  (CCB)  to  include 
the  user,  program  office,  development  contractor,  integration  contractor 
or  agency,  and  any  other  critical  stakeholder, 
o  For  legacy  systems,  the  CCB  shall  include  the  appropriate  support  and 
sustainment  organizations. 
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5.2.6.3  Review  of  Software-Intensive  Programs 

An  independent  expert  review  team  shall  review  programs  and  report  on  technology 
and  development  risk,  cost,  schedule,  design,  development,  project  management 
processes  and  the  application  of  systems  and  software  engineering  best  practices.  The 
team  shall  report  their  findings  directly  to  the  PM  and  the  PEO  or  equivalent 
management  official.  The  Deputy  Under  Secretary  of  Defense  for  Science  and 
Technology  shall  manage  the  team,  composed  of  a  small  group  of  software  systems 
engineering  and  technology  experts. 

|  DoD  Directive  8320.1 ,  DoD  Data  Administration,  September  26, 1991 
"  DoD  Directive  S-3600.1,  Information  Operations,  December9,  1996 
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THE  UNDER  SECRETARY  OF  DEFENSE 

3010  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3010 

2  1  DEC  2000 

MEMORANDUM  FOR  SERVICE  ACQUISITION  EXECUTIVES 

SUBJECT:  Independent  Expert  Program  Reviews  of  Software  Intensive  System  Acquisitions 

The  primary  recommendation  from  the  recent  Defense  Science  Board  study  on 
Defense  Software  was  the  institutionalization  of  Independent  Expert  Program  Reviews 
(lEPRs)  of  ACAT  I-III  software  intensive  systems.  Such  reviews  are  essential  to  help  the 
Program  Manager  adequately  address  issues  of  cost,  schedule,  technology,  risk  and  process, 
and  provide  an  opportunity  to  share  scarce  expert  resources.  They  are  most  effective  when 
performed  at  a  time  in  the  acquisition  lifecycle  when  they  can  prevent  problems,  well  before 
a  milestone  review,  rather  than  react  to  them.  The  Software  Intensive  Systems  Steering 
Group  (SISSG),  chaired  by  Deputy  Under  Secretary  of  Defense  (Science  and  Technology) 
(DUSD(S&T))  all  representing  the  senior  software  executives  from  each  of  the  Services  and 
ASD  C3I,  has  enthusiastically  endorsed  this  recommendation. 

It  is  important  to  implement  IEPRs  within  your  Service  such  that  they  support  your 
individual  programs  and  integrate  with  your  acquisition  processes  and  organizational 
responsibilities.  It  is  of  equal  importance  that  the  results  of  these  reviews  contribute  to  the 
greater  understanding  of  the  risks,  problems  and  best  practices  across  DoD.  Therefore,  I  am 
asking  the  DUSD(S&T)  to  establish  an  IEPR  Working  Group  (WG)  chaired  by  the  Director. 
Software  Intensive  Systems.  This  group  will  be  made  up  of  Service  and  OSD  software 
acquisition  specialists  and  representatives  from  assessment  organizations  throughout  DoD. 
The  IEPR  WG  will  develop  a  plan  and  necessary  policy  to  implement  IEPRs,  coordinate 
IEPR  activities,  promote  IEPR  consistency,  and  coordinate  the  collection,  analysis  and 
distribution  of  non-atiributable  IEPR  information.  The  SISSG,  as  a  conduit,  to  the  Services’ 
acquisition  community,  will  provide  vision  and  guidance  to  this  group.  The  DUSD(S&T) 
will  approve  the  IEPR  implementation  plan,  and  forward  the  plan  to  my  office  by  April  1, 
2001, 

Please  designate  your  IEPR  WG  representatives  by  January  5,  2001.  The  point  of 
contact  is  Dr.  Jack  Ferguson,  Director  of  Software  Intensive  Systems,  (203)  602-085  1,  ext. 
105. 


o 
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Charter  for  the  DoD  Software  Intensive  Systems  Steering  Group 


1.0  Introduction: 

1.1  Purpose  of  this  document.  This  document  establishes  the  charter  for  the  DoD  Software  Intensive 
Systems  Steering  Group  (SISSG).  The  charter  defines  the  purpose,  responsibilities,  operation,  meeting 
frequency  and  membership  of  the  SISSG  to  oversee  activities  related  to  improving  the  acquisition, 
development  and  sustainment  of  DoD  software  intensive  systems. 

1.2  Background.  Software  in  our  weapon  systems  provides  not  only  current  functionality  but  also  the 
flexibility  to  respond  to  changing  threats  and  environments  for  the  long  lifetimes  these  systems  must  endure. 
For  this  reason,  software  is  greatly  increasing  in  its  percentage  of  weapon  system  development  cost  and 
schedule.  It  is  estimated  that  approximately  40%  of  our  S37.6B  RDT&E  budget  is  spent  on  software. 
However,  we  consistently  endure  software  development/integration/improvement  cost  and  schedule 
overruns,  with  attendant  reductions  in  performance.  The  Under  Secretary  of  Defense  (Acquisition, 
Technology  and  Logistics)  and  the  Service  Acquisition  Executives  established  the  Software  Intensive 
Systems  Steering  Group  to  improve  the  acquisition,  development  and  sustainment  of  DoD  software 
intensive  systems. 

2.0  Purpose  of  the  Steering  Group: 

The  purpose  of  the  SISSG  is  to  bring  the  software  engineering  capabilities  of  DoD  and  the  military 
services  to  bear  on  collaborative  efforts  to  improve  the  development,  acquisition,  and  sustainment  of 
software  intensive  systems;  The  SISSG  will  establish  strategic  direction,  oversee  the  DoD  Software 
Intensive  System  effort,  and  prioritize  joint  activities  that  have  significant  improvement  potential. 

3.0  SG  Responsibilities: 

3.1  Establish  strategic  direction  for  improvements  in  software  intensive  system  acquisition,  development 
and  sustainment. 

3.2  Review,  evaluate  and  oversee  implementation  and/or  response  to  recommendations  of  Defense  Science 
Board  Task  Forces,  service  advisory  boards,  and  other  initiatives  related  to  DoD  software  and  provide 
expertise  for  such  efforts. 

3.3  Identify  and  prioritize  science  and  technology  requirements  and  opportunities  with  potential  for 
improving  software  intensive  system  acquisition,  development,  fielding,  and  sustainment. 

3.4  Identify  and  support  opportunities  to  transition  technologies  with  potential  for  improving  software 
intensive  system  acquisition,  development,  fielding,  and  sustainment. 

3.5  Identify  and  support  opportunities  to  improve  software  aspects  of  Advanced  Concept  Technology 
Demonstrations. 

3.6  Identify,  promote  and  oversee  efforts  to  improve  the  abilities  of  the  acquisition  workforce  to  manage 
software  intensive  system  acquisition. 

3.7  Identify,  promote  and  oversee  efforts  to  improve  the  national  software  development  and  sustainment 
infrastructure,  to  include  information  assurance. 

4.0  Meeting  Frequency: 

The  SISSG  will  initially  meet  monthly,  with  special  meetings  held  as  necessary.  Attendance  at  the 
SISSG  meetings  will  be  by  invitation  of  the  SISSG. 
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5.0  Changes  to  This  Charter: 

The  charter  may  be  updated  based  on  need.  Changes  to  the  charter  will  be  coordinated  with 
DUSD(S&T). 

6.0  Membership: 

6.1  Steering  Group  members.  The  current  composition  of  the  SISSG  will  be  as  follows: 


Organization 

Name 

Position 

DUSD(S&T) 

Delores  M.  Etter 

Chair 

Deputy  Under  Secretary  of  Defense 
(Science  and  Technology) 

Army 

Henry  C.  Dubin 

Director  for  Assessment  and  Evaluation, 
Office  of  the  Assistant  Secretary  of  the  Army 
(Acquisition,  Logistics  and  Technology) 

Navy 

Michael  J.  O’ Driscoll 

Assistant  Secretary  of  the  Navy  (Research, 
Development  and  Acquisition) 

Deputy  Chief  Engineer 

Air  Force 

Don  Daniel 

Deputy  Assistant  Secretary  of  the  Air  Force 
(Science,  Technology  and  Engineering) 

ASD(C3I) 

Margaret  Myers 

Principal  Director,  DASD  (Deputy  Chief 
Information  Officer) 

DUSD(S&T)/SIS 

Jack  R.  Ferguson 

Executive  Secretary 

Director  for  Software  Intensive  Systems, 
ODUSD(S&T) 

6.2  The  Steering  Group  will  be  supported  by  advisors,  such  as  DARPA  (Directors  of  Information  Technology 
Office  and  Information  Systems  Office)  and  the  Director  of  the  Software  Engineering  Institute. 
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The  following  are  GAO’s  comments  on  the  Department  of  Defense’s  letter 
dated  March  2,  2001. 


QAO  Comments  l-  We  disagree.  Sufficient  bases  currently  exist  for  issuance  of  a  DOD  SPI 

policy  requirement,  especially  in  light  of  the  evidence  in  our  report  that 
(1)  without  this  requirement  not  all  components  are  implementing  SPI 
and  (2)  those  components  that  are  currently  implementing  SPI  are 
reporting  substantial  benefits.  Nevertheless,  DOD's  decision  to  await 
an  OSD  work  group's  results  before  making  a  decision  on  how  to 
proceed  is  not  unreasonable  or  inconsistent  with  our  position. 

2.  See  response  to  comment  1. 


3.  We  disagree.  Oversight  is  an  important  part  of  policy  implementation, 
and  without  such  oversight,  DOD  would  incur  significant  risk  that  the 
policy  would  not  be  implemented.  Further,  establishing  a  baseline 
measure  to  determine  compliance  does  not  require  the  implementation 
of  a  specific  model.  The  intent  of  our  recommendations  is  to  establish  a 
policy  requiring  SPI  that  recognizes,  as  our  report  recognizes,  that  there 
is  more  than  one  model  for  doing  so  effectively. 
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Since  1984,  the  Software  Engineering  Institute  (SEI)  has  worked  to 
improve  management  of  software/systems  productivity  and  quality 
primarily  by  addressing  problems  in  acquiring,  developing,  engineering,  or 
enhancing  software/systems  through  a  series  of  capability  maturity  models. 
According  to  SEI,  an  organization’s  process  capability  provides  a  means  of 
predicting  the  most  likely  outcome  of  the  next  software/systems  project 
undertaken;  process  maturity  implies  that  the  productivity  and  quality 
resulting  from  an  organization’s  software/systems  processes  can  be 
improved  as  maturity  of  the  processes  increases.  The  IDEAL™  model  is 
based  on  lessons  learned  from  SEI  experiences  as  well  as  from  SEI  projects 
relating  to  software  process  capability  and  maturity.  For  example,  during 
the  initiating  phase  of  the  IDEAL™  model,  general  SPI  program  goals  are 
defined,  and  this  definition  could  be  in  terms  of  capability  maturity  model 
levels.  In  the  diagnosing  phase,  IDEAL™  recommends  developing  an 
organization  process  maturity  baseline;  SEI’s  capability  maturity  model- 
based  appraisal  is  one  way  of  establishing  this  baseline. 

The  first  of  these  capability  maturity  models,  the  Software  Capability 
Maturity  Model®  (SW-CMM®),1  was  designed  to  assist  organizations  in 
improving  software  development  and  maintenance  processes.  In  this 
model,  software  process  maturity — ranked  from  a  low  of  level  1  to  a  high  of 
level  5 — serves  as  an  indicator  of  the  likely  range  of  software  cost, 
schedule,  and  quality  that  can  be  expected  to  be  achieved  by  projects 
developed  within  an  organization.  (See  figure  2.) 


'Capability  Maturity  Model  and  CMM  are  registered  in  the  U.S.  Patent  and  Trademark  Office. 
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Figure  2:  SW-CMM®  Levels  and  Descriptions 


Source:  SEI. 


Since  the  SW-CMM®  was  published,  SEI  has  developed  additional  models 
in  the  capability  maturity  series: 
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•  The  Software  Acquisition  CMM®  is  a  model  for  improving  the  software 
acquisition  process.  It  follows  the  same  five-level  architecture  as  the 
SW-CMM®  but  emphasizes  acquisition  issues  and  the  needs  of 
individuals  and  groups  planning  and  managing  software  acquisition 
activities. 

•  The  Systems  Engineering  CMM®  describes  the  essential  elements  of  an 
organization’s  systems  engineering  process  and  provides  a  reference  for 
comparing  actual  systems  engineering  practices  against  these  elements. 
The  model  addresses  the  process  aspects  of  systems  engineering  and 
the  product  development  portion  of  the  life  cycle.  This  model  was  a 
collaboration  of  several  organizations,  including  SEI. 

•  In  1997  a  team  led  by  DOD,  in  conjunction  with  SEI,  government,  and 
industry,  concentrated  on  developing  an  integrated  framework  for 
maturity  models  and  associated  products.  The  result  was  the  CMM 
Integration811  (CMMIsm),2  which  is  intended  to  provide  guidance  for 
improving  an  organization’s  processes  and  the  ability  to  manage  the 
development,  acquisition,  and  maintenance  of  products  and  services, 
while  reducing  the  redundancy  and  inconsistency  caused  by  using 
stand-alone  models. 

The  CMMIsm  combines  earlier  models  from  SEI  and  the  Electronic 
Industries  Alliance3  into  a  single  model  for  use  by  organizations  pursuing 
enterprise-wide  process  improvement.  However,  the  prototype  CMMIsm 
does  not  include  the  acquisition  features  of  the  SA-CMM®  because  the 
team  wanted  to  focus  first  on  the  development  process.  A  CMMI811  that 
includes  coverage  for  acquiring  software-intensive  systems  is  currently 
being  developed.  Additional  disciplines  may  also  be  covered.  Ultimately, 
the  CMMIsm  is  to  replace  the  models  that  have  been  its  starting  point. 


2CMM  Integration  and  CMMI  are  service  marks  of  Carnegie  Mellon  University. 

3The  Electronic  Industries  Alliance  is  a  trade  organization  representing  over  2,000 
companies  involved  in  the  design,  manufacture,  and  sale  of  electronic  parts,  components, 
assemblies,  and  systems  for  residential,  commercial,  industrial,  military,  and  aerospace  use. 
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Army  SPI  Program 


Background 


The  Army  has  assigned  responsibility  for  information  systems  to  the  Army 
Materiel  Command  (AMC).  Several  major  subcommands  function  under 
AMC.  Three  of  these  major  subcommands — the  Communications- 
Electronics  Command  (CECOM),  the  Aviation  and  Missile  Command 
(AMCOM),  and  the  Tank-Automotive  Command — are  responsible  for  the 
acquisition,  development,  engineering,  and  maintenance  of  information 
technology  for  the  Army.  We  reviewed  the  Army’s  SPI  activities  at  CECOM 
and  AMCOM. 

CECOM  has  assigned  responsibility  for  information  systems  to  its  Software 
Engineering  Center  (SEC).  The  center,  located  at  Fort  Monmouth,  New 
Jersey,  is  supported  by  several  software/systems  activities  located  across 
the  United  States.  The  center  is  responsible  for  overseeing  about  85 
percent  of  the  Army’s  systems,  including  (1)  command,  control, 
communications,  and  computers;  (2)  intelligence,  electronic  warfare,  and 
sensors;  (3)  sustaining  base/power  projection;  and  (4)  AMC  business 
systems. 

AMCOM  has  assigned  responsibility  for  its  information  systems  to  its 
Software  Engineering  Directorate  (SED).  This  directorate,  located  at 
Redstone  Arsenal,  Alabama,  oversees  and  provides  life-cycle  support  to 
both  aviation  and  missile  weapons  systems.  (See  figure  3.) 


The  Army  depends  on  software-intensive  systems  to  support  each  of  its 
major  commands  and  every  facet  of  its  mission,  from  weapons  to  financial 
management  systems.  The  Army  budgeted  about  $3.3  billion  on 
information  technology  during  fiscal  year  2000. 
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Figure  3:  Partial  Army  Organization  Chart  Highlighting  Units  Responsible  for  Software/Systems 


n  DOD  component  or  software/ 
systems  unit  selected  for  review 

Source:  GAO  based  on  Army  data. 

Army’s  SPI  program  activities  began  in  the  early  1990s;  in  mid-1996  the 
Secretary  mandated  that  all  Army  software  acquisition,  development,  and 
maintenance  activities  establish  SPI  programs.  At  the  same  time,  the  Army 
published  an  SPI  policy1  that  specified  two  requirements: 

•  First,  a  contractor’s  capability  to  produce  quality  software  will  be  part  of 
the  Army’s  source-selection  evaluation  process.  The  Army  has 
implemented  this  requirement  by  evaluating  potential  contractors 
against  SW-CMM®  level  3  maturity  requirements  and  requiring 
contractors  that  do  not  meet  these  requirements  to  propose  a  strategy 
for  mitigating  the  risks  associated  with  not  meeting  them.  This 
requirement  is  further  enforced  during  milestone  reviews  of  major 


‘Army’s  SPI  policy  is  now  part  of  Army  Regulation  70-1. 
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systems,  when  the  program  manager  must  show  that  the  contractor 
meets  these  requirements. 

•  Second,  Army  software  activities  will  continually  improve  their  software 
and  systems  process  maturity,  including  self-assessments  of  existing 
processes,  and  achieve  SW-CMM®  level  3  within  6  years  of  initial 
assessment. 


Army’s  SPI  Program  Is 
Aligned  With  SEI’s  IDEALSM 
Model 

Both  the  CECOM  SEC  and  the  AMCOM  SED  SPI  programs  are  fully 
consistent  with  the  IDEALSM  model.  Table  5  shows  examples  of  program 
elements  that  reflect  some  of  the  recommended  tasks  in  the  IDEAL™ 
model;  table  6  provides  a  detailed  comparison  of  CECOM  and  AMCOM’s 

SPI  programs  against  each  of  the  IDEAL™  model  recommended  tasks. 

Table  5:  Army  Examples  of  Alignment  With  IDEALSM 

Phase/tasks 

Task  example 

Initiating:  Define  General  SPI  goals 

Army  issued  a  1996  policy  that  requires  all  Army  software  activities  to  continually  improve 
their  software  and  systems  process  maturity,  including  performing  self-assessments  of 
existing  processes,  and  achieving  SW-CMM®  level  3  within  6  years  of  initial  assessment. 

Diagnosing:  Determine  what  baseline(s) 
are  needed 

Both  CECOM’s  SEC  and  AMCOM’s  SED  have,  as  part  of  their  continuous  process, 
established  SEPGs  that  are  constantly  reviewing  baselines  and  making  changes  as 
needed. 

Establishing:  Create  and  then  update  an 
SPI  strategic  plan 

Army’s  latest  updated  strategic  plan,  which  addresses  SPI,  was  issued  in  1997;  CECOM 
has  its  own  strategic  plan  that  also  addresses  SPI,  which  was  last  revised  in  1998. 

Acting:  Transition  to  long-term  support 

One  way  to  transition  to  support  is  to  implement  policies  and  handbooks  that  software 
activities  can  use  as  guidance  to  improve.  CECOM  issued  software  policy  in  1996. 

AMCOM  issued  a  software  engineering  process  handbook  and  a  procedures  and 
standards  handbook  in  1993;  these  two  publications  were  combined  into  one  in  1998. 

Leveraging:  Gather  and  analyze  lessons 
learned,  and  revise  the  organizational 
approach,  if  necessary 

Both  CECOM  and  AMCOM  established  SEPGs;  they  gather  information  on  SPI  quality  at 
their  respective  commands  and  meet  weekly  to  review  what  they  have  learned  and,  if 
needed,  reestablish  goals. 
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Table  6:  Comparisons  of  Army  SPI  Activities  With  the  IDEALSM  Model 


Satisfied? 

Phase 

Task 

AMCOM 

SED 

CECOM 

SEC 

Initiating 

Organize  discovery  team  to  develop  a  proposal  to  management  for  launching 
SPI  program 

Yes 

Yes 

Identify  business  needs  and  drivers  for  improvement 

Yes 

Yes 

Build  an  SPI  proposal 

Yes 

Yes 

Educate  and  build  support 

Yes 

Yes 

Obtain  approval  for  SPI  proposal  and  initial  resources 

Yes 

Yes 

Establish  SPI  infrastructure 

Yes 

Yes 

Assess  the  climate  for  SPI 

Yes 

Yes 

Define  general  SPI  goals 

Yes 

Yes 

Define  guiding  principles  of  SPI  program 

Yes 

Yes 

Launch  the  program 

Yes 

Yes 

Diagnosing 

Determine  what  baseline(s)  are  needed 

Yes 

Yes 

Plan  for  baseline(s) 

Yes 

Yes 

Conduct  baseline(s) 

Yes 

Yes 

Present  findings 

Yes 

Yes 

Develop  final  findings  and  recommendations  report 

Yes 

Yes 

Communicate  findings  and  recommendations  to  organization 

Yes 

Yes 

Establishing 

Select  and  get  training  in  a  strategic  planning  process 

Yes 

Yes 

Review  organization’s  vision 

Yes 

Yes 

Review  organization’s  business  plan 

Yes 

Yes 

Determine  key  business  issues 

Yes 

Yes 

Review  past  improvement  efforts 

Yes 

Yes 

Describe  motivations  to  improve 

Yes 

Yes 

Identify  current  and  future  (planned)  improvement  efforts 

Yes 

Yes 

Finalize  roles  and  responsibilities  of  infrastructure  entities 

Yes 

Yes 

Prioritize  activities  and  develop  improvement  agenda 

Yes 

Yes 

Reconcile  existing  planned  improvement  efforts  with  baseline  findings  and 
recommendations 

Yes 

Yes 

Transform  general  SPI  goals  to  measurable  goals 

Yes 

Yes 

Create/update  SPI  strategic  plan 

Yes 

Yes 

Build  consensus,  review,  approve  SPI  strategic  plan  and  commit  resources 

Yes 

Yes 

Form  technical  working  group 

Yes 

Yes 
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Satisfied? 

Phase 

Task 

AMCOM 

SED 

CECOM 

SEC 

Acting 

Complete  tactical  plan  for  technical  working  group 

Yes 

Yes 

Develop  solutions 

Yes 

Yes 

Pilot  potential  solutions 

Yes 

Yes 

Select  solution  providers 

Yes 

Yes 

Determine  long-term  support  needs 

Yes 

Yes 

Develop  rollout  strategy  and  plan  template 

Yes 

Yes 

Package  improvement  and  turn  over  to  SEPG 

Yes 

Yes 

Disband  technical  working  group 

Yes 

Yes 

Roll  out  solution 

Yes 

Yes 

Transition  to  long-term  support 

Yes 

Yes 

Leveraging 

Gather  lessons  learned 

Yes 

Yes 

Analyze  lessons  learned 

Yes 

Yes 

Revise  organizational  approach 

Yes 

Yes 

Review  sponsorship  and  commitment 

Yes 

Yes 

Establish-high  level  goals 

Yes 

Yes 

Develop  new/revised  SPI  proposal 

Yes 

Yes 

Continue  with  SPI 

Yes 

Yes 

Managing 

Set  the  stage  for  SPI 

Yes 

Yes 

Organize  the  SPI  program 

Yes 

Yes 

Plan  the  SPI  program 

Yes 

Yes 

Staff  the  SPI  program 

Yes 

Yes 

Monitor  the  SPI  program 

Yes 

Yes 

Direct  the  SPI  program 

Yes 

Yes 

When  the  Army  first  launched  its  SPI  activities,  it  managed  initiation  and 
diagnosis  centrally,  with  both  CECOM  and  AMCOM  being  involved  in  these 
early  actions.  Subsequently,  as  many  groups  throughout  the  Army  were 
trained  in  using  the  SEI  process  maturity  measurements,  responsibility  for 
implementing  SPI  programs  was  delegated  to  the  commands.  The  Army  has 
since  expanded  this  decentralized  approach,  giving  each  command  the  SPI 
requirements  through  Army  policy  and  allowing  each  to  implement  the 
policy  as  it  determines  best  supports  its  mission. 


Army  Reports  That  Its 
Decentralized  Approach  to 
SPI  Program  Management 
Has  Produced  Results 
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According  to  information  that  these  two  subcommands  provided,  their  SPI 
programs  have  produced  positive  results.  One  of  AMCOM’s  measures  of 
software  quality  is  development  productivity,  which  is  the  number  of  lines 
of  software  code  produced  as  a  function  of  resources  invested.  According 
to  AMCOM,  SED’s  productivity  ratio2  for  new  development  products 
increased  from  1.30  to  2.48  as  a  result  of  moving  from  SW-CMM®  level  2  to 
level  3.  SED  reports  that  it  has  recently  achieved  level  4. 


Air  Force  SPI  Program 


Background  Software-intensive  systems  are  vital  to  the  Air  Force’s  overall  mission. 

They  are  used  to  sustain  weapons  systems,  airborne  electronics,  electronic 
warfare,  space  communications,  and  support  equipment.  The  Air  Force  has 
about  1,600  systems  and  budgeted  about  $4.6  billion  in  fiscal  year  2000  for 
information  technology. 

The  Air  Force  has  nine  major  commands,  but  its  largest  software/systems 
units  are  under  the  Air  Force  Materiel  Command  (AFMC).  Within  AFMC, 
we  reviewed  SPI  efforts  at  two  units  within  the  Electronic  Systems  Center, 
which  provides  command  and  control  and  information  systems  for  the  Air 
Force  as  well  as  for  other  DOD  units,  using  a  budget  of  over  $3  billion  in 
fiscal  year  2000.  The  two  units  that  we  reviewed  were  the  Standard 
Systems  Group  (SSG)  at  Montgomery,  Alabama,  and  the  Materiel  Systems 
Group  (MSG)  at  Dayton,  Ohio.  In  addition,  we  reviewed  SPI  activities  at  the 
Air  Force  Academy  (AFA),  which  has  one  of  the  remaining 
software/systems  units  outside  AFMC.  (See  figure  4.) 


Productivity  equals  total  software  lines  of  code  developed  divided  by  the  total  effort 
expended. 
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Figure  4:  Partial  Air  Force  Organization  Chart  Highlighting  Units  Responsible  for  Software/Systems 


□  DOD  component  or  software/ 
systems  unit  selected  for  review 


Source:  GAO  based  on  Air  Force  data. 

SSG  is  the  largest  software/systems  unit  within  the  Air  Force  in  terms  of 
money  invested  and  amount  of  software  delivered.  Its  mission  is  to  develop 
and  maintain  combat  support  information  systems  for  the  Air  Force  and 
other  DOD  components.  Additionally,  SSG  manages  information 
technology  contracts  and  standard  information  systems  programs 
commonly  used  at  all  active  and  reserve  Air  Force  bases  and  some  DOD 
agencies  worldwide. 

Next  to  SSG,  MSG  is  the  largest  Air  Force  central  software/systems  unit. 
MSG’s  mission  is  to  support  the  Air  Force  goal  of  information  dominance 
through  acquiring,  developing,  maintaining,  reengineering,  and  providing 
technical  services  for  information  systems. 

AFA  has  a  software/systems  unit  that  is  primarily  responsible  for 
maintaining  and  developing  the  Cadet  Administrative  Management 
Information  System,  a  mission-critical  database  system  that  tracks  the 
progress  of  cadets  from  precandidacy  through  academic,  physical, 
ethical/moral,  and  military  training  programs  and,  after  graduation, 
throughout  their  Air  Force  careers. 
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In  1991,  the  Deputy  Assistant  Secretary  of  the  Air  Force  initiated  the 
service’s  SPI  program.  In  particular,  Air  Force  software/systems  units  were 
directed  to  complete  SW-CMM®  assessments  by  October  1,  1994,  perform 
follow-up  assessments  every  2  years,  and  achieve  SW-CMM®  level  3  by 
1998.  Air  Force’s  1994  SPI  policy  was  revised  this  year.3  This  revised  policy 
requires  all  units  that  develop  or  maintain  software/systems  to  have  an  SPI 
program  and  a  documented  SPI  plan  that  includes,  at  least,  a  baseline 
measure  of  their  current  capabilities,  goals  and  milestones  they  intend  to 
reach,  and  metrics  with  which  to  measure  their  progress  toward  goals  and 
milestones. 


Air  Force  SPI  Program  Is 
Aligned  With  SEI’s  IDEALSM 
Model 


The  IDEALSM  model  is  the  framework  the  Air  Force  recommends  to  its 
software/systems  units,  and  our  comparison  of  the  activities  at  SSG,  MSG, 
and  AFA  to  the  IDEALSM  model  found  that  their  respective  SPI  programs 
are  almost  all  aligned  with  the  model.  Specifically,  each  of  the  programs 
satisfied  all  but  five  of  the  IDEALSM  model  recommended  tasks,  and  none  of 
those  five  is  significant  enough  to  preclude  having  effective  SPI  programs. 
Table  7  shows  examples  of  the  programs’  elements  that  reflect  some  of  the 
recommended  tasks  in  the  IDEALSM  model;  table  8  shows  a  detailed 
comparison  of  SSG,  MSG,  and  AFA  SPI  programs  against  each  of  the 
IDEALsm  model  recommended  tasks. 


Table  7:  Air  Force  Examples  of  Alignment  With  IDEALSM 

Phase/tasks 

Task  example 

Initiating:  Establish  SPI  infrastructure 

In  1 993,  AFA  completed  a  self-assessment  of  software  engineering  processes,  identifying 
key  areas  for  improvement,  establishing  an  SPI  program,  and  assigning  responsibility  for 
it. 

Diagnosing:  Appraise,  characterize,  and 
assess  process 

By  1996,  all  41  Air  Force  software  units  had  completed  their  initial  CMM®  assessments, 

21  systems  had  conducted  a  second  assessment,  and  eight  were  conducting  a  third 
assessment. 

Establishing:  Strategize,  set  priorities 

AFMC  developed  a  strategic  plan  for  SSG  and  MSG,  prioritized  activities,  and  provided  an 
improvement  agenda. 

Acting:  Execute  planned  improvements 

Based  on  its  experience  with  a  variety  of  SPI  projects,  SSG  developed  and  is 
implementing  a  standard  software  development  process  for  all  software  projects  within 

SSG,  regardless  of  project  type. 

3Air  Force  Instruction  33-114,  July  1,  2000. 


Page  45 


GAO-Ol-116  Defense  Information  Technology 


Appendix  IV 

Detailed  Results  of  Review  of  DOD 
Components’  SPI  Programs 


(Continued  From  Previous  Page) 

Phase/tasks  Task  example 

Leveraging:  Document  and  analyze  lessons  SSG  shares  benchmarking  processes  with  MSG  in  a  strategic  partnership;  MSG 
learned,  plan  changes  for  next  cycle  documents  lessons  learned  and  enters  them  into  a  database. 


Table  8:  Comparisons  of  Air  Force  SPI  Activities  With  the  IDEALSM  Model 


Phase 

Task 

AFA 

Satisfied? 

SSG 

MSG 

Initiating 

Organize  discovery  team  to  develop  a  proposal  to  management  for 
launching  SPI  program 

Yes 

Yes 

Yes 

Identify  business  needs  and  drivers  for  improvement 

Yes 

Yes 

Yes 

Build  an  SPI  proposal 

Yes 

Yes 

Yes 

Educate  and  build  support 

Yes 

Yes 

Yes 

Obtain  approval  for  SPI  proposal  and  initial  resources 

Yes 

Yes 

Yes 

Establish  SPI  infrastructure 

Yes 

Yes 

Yes 

Assess  the  climate  for  SPI 

No 

No 

No 

Define  general  SPI  goals 

Yes 

Yes 

Yes 

Define  guiding  principles  of  SPI  program 

Yes 

Yes 

Yes 

Launch  the  program 

Yes 

Yes 

Yes 

Diagnosing 

Determine  what  baseline(s)  are  needed 

Yes 

Yes 

Yes 

Plan  for  baseline(s) 

Yes 

Yes 

Yes 

Conduct  baseline(s) 

Yes 

Yes 

Yes 

Present  findings 

Yes 

Yes 

Yes 

Develop  final  findings  and  recommendations  report 

Yes 

Yes 

Yes 

Communicate  findings  and  recommendations  to  organization 

Yes 

Yes 

Yes 

Establishing 

Select  and  get  training  in  a  strategic  planning  process 

Yes 

Yes 

Yes 

Review  organization’s  vision 

Yes 

Yes 

Yes 

Review  organization’s  business  plan 

Yes 

Yes 

Yes 

Determine  key  business  issues 

Yes 

Yes 

Yes 

Review  past  improvement  efforts 

Yes 

Yes 

Yes 

Describe  motivations  to  improve 

Yes 

Yes 

Yes 

Identify  current  and  future  (planned)  improvement  efforts 

Yes 

Yes 

Yes 

Finalize  roles  and  responsibilities  of  infrastructure  entities 

Yes 

Yes 

Yes 

Prioritize  activities  and  develop  improvement  agenda 

Yes 

Yes 

Yes 

Reconcile  existing  planned  improvement  efforts  with  baseline  findings 
and  recommendations 

No 

No 

No 

Transform  general  SPI  goals  to  measurable  goals 

Yes 

Yes 

Yes 
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Satisfied? 

Phase 

Task 

AFA 

SSG 

MSG 

Create/update  SPI  strategic  plan 

Yes 

Yes 

Yes 

Build  consensus,  review,  approve  SPI  strategic  plan  and  commit 
resources 

Yes 

Yes 

Yes 

Form  technical  working  group 

Yes 

Yes 

Yes 

Acting 

Complete  tactical  plan  for  technical  working  group 

Yes 

Yes 

Yes 

Develop  solutions 

Yes 

Yes 

Yes 

Pilot  potential  solutions 

Yes 

Yes 

Yes 

Select  solution  providers 

Yes 

Yes 

Yes 

Determine  long-term  support  needs 

Yes 

Yes 

Yes 

Develop  rollout  strategy  and  plan  template 

Yes 

Yes 

Yes 

Package  improvement  and  turn  over  to  SEPG 

Yes 

Yes 

Yes 

Disband  technical  working  group 

No 

No 

No 

Rollout  solution 

Yes 

Yes 

Yes 

Transition  to  long-term  support 

No 

No 

No 

Leveraging 

Gather  lessons  learned 

Yes 

Yes 

Yes 

Analyze  lessons  learned 

Yes 

Yes 

Yes 

Revise  organizational  approach 

Yes 

Yes 

Yes 

Review  sponsorship  and  commitment 

No 

No 

No 

Establish  high-level  goals 

Yes 

Yes 

Yes 

Develop  new/revised  SPI  proposal 

Yes 

Yes 

Yes 

Continue  with  SPI 

Yes 

Yes 

Yes 

Managing 

Set  the  stage  for  SPI 

Yes 

Yes 

Yes 

Organize  the  SPI  program 

Yes 

Yes 

Yes 

Plan  the  SPI  program 

Yes 

Yes 

Yes 

Staff  the  SPI  program 

Yes 

Yes 

Yes 

Monitor  the  SPI  program 

Yes 

Yes 

Yes 

Direct  the  SPI  program 

Yes 

Yes 

Yes 

Air  Force  Reports  That  Its 
Decentralized  Approach  to 
SPI  Program  Management 
Has  Produced  Results 


Air  Force  headquarters  has  delegated  SPI  responsibility  to  its 
software/systems  units.  When  the  Air  Force  began  its  SPI  activities  in  1991, 
its  goal  was  to  initiate  SPI  by  performing  assessments  that  would  indicate 
the  current  level  of  maturity  at  Air  Force  units.  Management  of  this  effort 
was  centralized  in  the  Air  Force  Communications  Agency  (AFCA).  AFCA 
staff  visited  all  41  Air  Force  units,  some  more  than  once,  to  perform 
assessments.  Once  software/systems  units  became  capable  of  conducting 
their  own  process  maturity  measurements,  Air  Force  began  decentralizing 
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management  of  the  SPI  program  to  the  units.  The  last  year  in  which  the  Air 
Force  exercised  any  centralized  management  of  SPI  was  1998. 

The  Air  Force’s  SPI  efforts  have  been,  in  its  view,  beneficial.  For  example, 
one  Air  Force  center  reported  a  7.5-to-l  return  on  its  SPI  investment,  which 
was  independently  verified.  An  official  at  another  center  stated  that  SPI 
had  allowed  its  organization  to  achieve  higher  process  maturity  levels  and 
made  significant  improvements  in  the  quality  of  its  software  products  and 
its  productivity  measures. 


Navy  SPI  Program 


Background  The  Navy  depends  on  software-intensive  systems  to  support  many 

functions  throughout  its  nine  operating  forces — including  the  Marine 
Corps — and  its  15  support  units — including  four  major  systems  commands. 
These  systems  support  some  aspect  of  every  operation,  including  strategic 
and  tactical  operations;  sophisticated  weaponry;  intelligence,  surveillance, 
and  security;  strategic  sealift  and  fleet  mobilization  and  readiness;  and 
routine  business  functions  such  as  finance,  personnel,  logistics,  and 
contract  management.  In  fiscal  year  2000,  the  Navy  budgeted  about 
$3.1  billion  for  information  technology. 

Within  the  Navy,  acquisition,  development,  and  maintenance  of  these 
systems  is  delegated  to  its  major  systems  commands:  the  Naval  Aviation 
Systems  Command  (NAVAIR),  Space  and  Naval  Warfare  Systems  Command 
(SPAWAR),  Naval  Sea  Systems  Command,  and  Naval  Supply  Systems 
Command.  We  reviewed  SPI  activities  at  NAVAIR  and  SPAWAR.  Both 
commands  have  several  subordinate  units  involved  in  acquiring, 
developing,  and  maintaining  systems.  (See  figure  5.) 
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Figure  5:  Partial  Navy  Organization  Chart  Highlighting  Units  Responsible  for  Software/Systems 


□  DOD  component  or  software/ 
systems  unit  selected  for  review 

Source:  GAO  based  on  Navy  data. 

NAVAIR  provides  full  life-cycle  support  to  148  programs,  such  as  aircraft, 
avionics,  air-launched  weapons,  electronic  warfare,  cruise  missiles,  and 
unmanned  aerial  vehicles.  NAVAIR  has  two  divisions  (weapons  and 
aircraft).  The  weapons  division  has  two  California  product  centers,  and  the 
aircraft  division  has  three  centers,  located  in  New  Jersey,  Maryland,  and 
Florida. 
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SPAWAR  develops,  acquires,  and  maintains  systems  through  three  SPAWAR 
Systems  Centers  (SSC).  These  centers  are  at  San  Diego,  California; 
Chesapeake,  Virginia;  and  Charleston,  South  Carolina.  We  reviewed 
SPAWAR’s  SPI  efforts  at  all  three  centers.  SSC  San  Diego  develops, 
acquires,  and  supports  command,  control,  communications,  and  ocean 
surveillance  systems.  SSC  Chesapeake  develops,  acquires,  and  supports 
supply,  inventory,  finance,  food  service,  and  other  information  systems. 
SSC  Charleston  develops,  acquires,  and  supports  command,  control, 
communications,  intelligence,  surveillance,  and  reconnaissance  systems. 

To  guide  and  direct  their  respective  SPI  programs,  these  commands  follow 
DOD  and  other  models  and  standards.4  Commands  have  also  established 
local  policy.  For  instance,  SPAWAR  policy  requires  all  managers  with 
software-related  responsibilities  at  San  Diego  to  incorporate  process 
improvement  in  the  areas  of  new  development,  modification,  reuse, 
reengineering,  maintenance,  integration,  and  all  other  activities  resulting  in 
software  products.  In  2000,  NAVAIR  published  an  interim  policy  that 
requires  prospective  contractors  to  be  evaluated  at  SEI  SW-CMM®  level  3 
for  all  acquisitions. 


Navy’s  SPI  Program  Is  Partly  Navy’s  experience  with  SPI  to  date  has  been  mixed.  Both  SSC  San  Diego 
Aligned  With  the  IDEALSM  ami  NAVAIR  have  SPI  programs  that  are  consistent  with  the  IDEALSM 
Model  model.  However,  SSC  Chesapeake’s  and  SSC  Charleston’s  programs  are 

not.  Specifically,  SSC  Chesapeake  has  only  recently  initiated  an  SPI 
program  and,  while  efforts  to  date  are  aligned  with  the  IDEAL™  model, 
many  important  SPI  program  tasks  have  yet  to  be  executed.  For  example, 
in  July  2000  it  completed  some  initiating-phase  tasks,  such  as  creating  a 
management  steering  group  and  an  SEPG.  However,  it  has  yet,  for  example, 
to  (1)  conduct  baselines  to  identify  process  strengths  and  weaknesses  in 
the  diagnosing  phase,  (2)  develop  an  SPI  plan  with  measurable  goals  and 
committed  resources  in  the  establishing  phase,  (3)  pilot-test  potential 
solutions  or  transition  the  solutions  to  long-term  support  in  the  acting 
phase,  or  (4)  gather  or  analyze  lessons  learned  in  the  leveraging  phase. 


4The  Navy  uses  guidance  from  DOD  Directive  5000.1  and  DOD  Regulation  5000. 2-R,  SEI,  the 
DOD  Software  Program  Managers  Network’s  16  Critical  Software  Practices,  and  the 
Institute  of  Electrical  and  Electronics  Engineers/Electronic  Industries  Alliance  Standard 
12207. 
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In  the  case  of  SSC  Charleston,  no  SPI  program  exists,  although  the  center 
has  undertaken  one  task  that  is  intended  to  begin  the  initiating  phase  of  a 
program.  Table  9  shows  examples  of  Navy  SPI  programs’  elements  that 
reflect  some  of  the  recommended  tasks  in  the  IDEALSM  model;  table  10 
shows  a  detailed  comparison  of  NAVAIR  and  SPAWAR  SPI  programs 
against  each  of  the  IDEALSM  model  recommended  tasks. 


Table  9:  Examples  of  Navy  Alignment  With  IDEALSM 


Phase/tasks 

Task  example 

Initiating:  Identify  business  needs  and  drivers 
for  improvement 

SSC  San  Diego  identified  its  key  elements  for  project  success  in  three  broad  areas — 
process,  people,  and  technology — by  conducting  research  on  process  improvement 
traits  of  other  successful  organizations  and  contracting  with  SEI  to  identify  program 
weaknesses  and  key  areas  for  improvement. 

Diagnosing:  Plan  for  and  conduct  baseline 
activities 

SSC  San  Diego  developed  a  plan  for  establishing  a  baseline  for  all  software  projects,  and 
all  new  projects  are  baselined  and  assessed  before  they  are  implemented. 

Establishing:  Finalize  roles  and 
responsibilities  of  infrastructure  entities 

NAVAIR’s  plan  for  process  improvement  identifies  the  general  roles  and  responsibilities 
in  the  program.  The  Software  Process  Improvement  Office  has  a  formal  charter  that 
identifies  specific  roles,  goals,  and  responsibilities. 

Acting:  Pilot-test  potential  solutions 

SSC  San  Diego  pilot-tested  18  SPI  projects  with  over  400  staff  from  six  divisions  to  raise 
CMM®  maturity  levels. 

Leveraging:  Analyze  lessons  learned 

SSC  San  Diego  requires  that  all  projects  record  lessons-learned  data,  which  are  fed  into 
a  database  that  is  tracked,  reported,  and  shared  across  the  organization  at  two  levels  of 
best  practice — organizational  and  project. 

Table  10:  Comparisons  of  Navy  SPI  Activities  With  the  IDEALSM  Model 


Satisfied? 

Phase 

Task  NAVAIR 

SSC 

San  Diego 

SSC 

Chesapeake 

SSC 

Charleston 

Initiating 

Organize  discovery  team  to  develop  a  proposal  Yes 
to  management  for  launching  SPI  program 

Yes 

Yes 

Yes 

Identify  business  needs  and  drivers  for 
improvement 

Yes 

Yes 

Yes 

No 

Build  an  SPI  proposal 

Yes 

Yes 

Yes 

No 

Educate  and  build  support 

Yes 

Yes 

Yes 

No 

Obtain  approval  for  SPI  proposal  and  initial 
resources 

Yes 

Yes 

Yes 

No 
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(Continued  From  Previous  Page) 

Satisfied? 

Phase 

Task 

NAVAIR 

ssc 

San  Diego 

SSC 

Chesapeake 

SSC 

Charleston 

Establish  SPI  infrastructure 

Yes 

Yes 

Yes 

No 

Assess  the  climate  for  software  process 
improvement 

Yes 

Yes 

Yes 

No 

Define  general  SPI  goals 

Yes 

Yes 

Yes 

No 

Define  guiding  principles  of  SPI  program 

Yes 

Yes 

Yes 

No 

Launch  the  program 

Yes 

Yes 

Yes 

No 

Diagnosing 

Determine  what  baseline(s)  are  needed 

Yes 

Yes 

No 

No 

Plan  for  baseline(s) 

Yes 

Yes 

No 

No 

Conduct  baseline(s) 

Yes 

Yes 

No 

No 

Present  findings 

Yes 

Yes 

No 

No 

Develop  final  findings  and  recommendations 
report 

Yes 

Yes 

No 

No 

Communicate  findings  and  recommendations 
to  organization 

Yes 

Yes 

No 

No 

Establishing 

Select  and  get  training  in  a  strategic  planning 
process 

Yes 

Yes 

No 

No 

Review  organization’s  vision 

Yes 

Yes 

No 

No 

Review  organization’s  business  plan 

Yes 

Yes 

No 

No 

Determine  key  business  issues 

Yes 

Yes 

No 

No 

Review  past  improvement  efforts 

Yes 

Yes 

No 

No 

Describe  motivations  to  improve 

Yes 

Yes 

No 

No 

Identify  current  and  future  (planned) 
improvement  efforts 

Yes 

Yes 

No 

No 

Finalize  roles  and  responsibilities  of 
infrastructure  entities 

Yes 

Yes 

No 

No 

Prioritize  activities  and  develop  improvement 
agenda 

Yes 

Yes 

No 

No 

Reconcile  existing  planned  improvement 
efforts  with  baseline  findings  and 
recommendations 

Yes 

Yes 

No 

No 

Transform  general  SPI  goals  to  measurable 
goals 

Yes 

Yes 

No 

No 

Create/update  SPI  strategic  plan 

Yes 

Yes 

No 

No 

Build  consensus,  review,  approve  SPI  strategic 
plan  and  commit  resources 

Yes 

Yes 

No 

No 

Form  technical  working  group 

Yes 

Yes 

No 

No 
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(Continued  From  Previous  Page) 


Satisfied? 


Phase 

Task 

NAVAIR 

ssc 

San  Diego 

SSC 

Chesapeake 

SSC 

Charleston 

Acting 

Complete  tactical  plan  for  technical  working 
group 

Yes 

Yes 

No 

No 

Develop  solutions 

Yes 

Yes 

No 

No 

Pilot  potential  solutions 

Yes 

Yes 

No 

No 

Select  solution  providers 

Yes 

Yes 

No 

No 

Determine  long-term  support  needs 

Yes 

Yes 

No 

No 

Develop  rollout  strategy  and  plan  template 

Yes 

Yes 

No 

No 

Package  improvement  and  turn  over  to  SEPG 

Yes 

Yes 

No 

No 

Disband  technical  working  group 

Yes 

Yes 

No 

No 

Roll  out  solution 

Yes 

Yes 

No 

No 

Transition  to  long-term  support 

No 

Yes 

No 

No 

Leveraging 

Gather  lessons  learned 

Yes 

Yes 

No 

No 

Analyze  lessons  learned 

Yes 

Yes 

No 

No 

Revise  organizational  approach 

Yes 

Yes 

No 

No 

Review  sponsorship  and  commitment 

Yes 

Yes 

No 

No 

Establish  high-level  goals 

Yes 

Yes 

No 

No 

Develop  new/revised  SPI  proposal 

Yes 

Yes 

No 

No 

Continue  with  SPI 

Yes 

Yes 

No 

No 

Managing 

Set  the  stage  for  SPI 

Yes 

Yes 

Yes 

No 

Organize  the  SPI  program 

Yes 

Yes 

Yes 

No 

Plan  the  SPI  program 

Yes 

Yes 

No 

No 

Staff  the  SPI  program 

Yes 

Yes 

Yes 

No 

Monitor  the  SPI  program 

Yes 

Yes 

Yes 

No 

Direct  the  SPI  program 

Yes 

Yes 

Yes 

No 

The  Navy  has  delegated  SPI  responsibility  to  its  commands,  which  in  some 
cases  have  further  decentralized  SPI  program  management  within  the 
command  structure.  For  example,  NAVAIR  manages  its  SPI  program 
centrally  through  its  Software  Process  Improvement  Office.  Established  in 
1999,  this  office,  in  combination  with  two  NAVAIR  executive  groups, 
establishes  NAVAIR  software  improvement  policies,  monitors 
performance,  and  provides  support  for  process  training  and  baselining.  In 
contrast  to  NAVAIR,  SPAWAR  decentralized  SPI  program  management  to 
its  SSCs. 


Navy  Reports  That  Its 
Decentralized  Approach  to 
SPI  Program  Management 
Has  Produced  Results 
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Navy  reports  several  SPI  program  benefits.  For  example,  officials  at 
NAVAIR’s  F/A-18  software  program  report  reaching  SW-CMM®  level  3  with 
benefits  including  cost  savings,  improved  product  quality,  and  a  7:1  return 
on  their  SPI  investment.  In  addition,  SSC  San  Diego  officials  report  that 
their  SPI  program  significantly  reduced  both  the  number  of  software 
defects  and  the  time  expended  in  testing  their  air  traffic  control  program 
system.  In  particular,  staff  months  spent  addressing  trouble  reports  were 
reduced  by  70  percent.  These  officials  also  state  that  benefits  from  SPI 
include  better  management  control,  improved  overall  software 
performance,  higher  customer  satisfaction,  and  increased  competitive 
advantage  and  repeat  business. 


Marine  Corps  SPI 
Program 


Background  The  Marine  Corps  depends  on  software-intensive  systems  to  support  every 

facet  of  its  mission — from  weapons  to  tactical  communications  systems.  In 
fiscal  year  2000,  the  Marine  Corps  budgeted  about  $525  million  for 
information  technology. 

The  Marine  Corps  has  assigned  responsibility  for  acquisition,  development, 
engineering,  and  maintenance  of  information  technology  to  the  Marine 
Corps  Systems  Command.  The  Command  is  the  sole  procurement  activity 
for  the  Marine  Corps,  purchasing  everything  from  business  systems  to 
software-intensive  weaponry  such  as  tanks  and  command,  control, 
communications,  and  computer  equipment.  The  Marine  Corps  Tactical 
Systems  Support  Activity  (MCTSSA),  located  at  Camp  Pendleton, 
California,  is  a  subordinate  command.  MCTSSA  is  responsible  for  software 
life-cycle  support  of  designated  Marine  Corps  and  joint-service  tactical  data 
systems  and  software.  (See  figure  6.) 
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Figure  6:  Partial  Marine  Corps  Organization  Chart  Highlighting  Units  Responsible 
for  Software/Systems 


□  DOD  component  or  software/ 
systems  unit  selected  for  review 

Source:  GAO  based  on  Marine  Corps  data. 

According  to  MCTSSA  officials,  the  Marine  Corps  does  not  have  a  formal 
SPI  program,  although  it  has  performed  SPI  activities  since  the  early  1990s. 
MCTSSA  uses  both  DOD  and  Marine  Corps  guidance  to  manage  its  SPI 
activities.5  At  one  time,  however,  MCTSSA  appeared  to  be  on  its  way  to  a 
formal  SPI  program.  It  started  SPI  activities  in  the  early  1990s,  and  by  1995 
was  using  SEI  to  support  them.  For  example,  during  1995  and  1996  SEI 
assisted  the  Marine  Corps  in  identifying  program  weaknesses  and  in 
developing  solutions  to  improve  them.  However,  MCTSSA  officials  stated 
that  they  did  not  renew  the  SEI  contract  because  of  a  lack  of  funds. 


5DOD  Directive  5000.1,  DOD  Regulation  5000.2-R,  Marine  Corps  Order  5000.22,  and  Marine 
Corps  Activity  Orders  4130.3  and  4130.4. 
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Marine  Corps  SPI  Activities 
Are  Partly  Aligned  With 
SEI’s  IDEALSM  Model 


MCTSSA  is  performing  many  of  the  tasks  recommended  by  the  IDEALSM 
model.  Table  11  shows  examples  of  activities  that  reflect  some  of  the 
recommended  tasks  in  the  IDEALSM  model;  however,  in  all  but  the 
diagnosing  phase,  MCTSSA  is  not  executing  some  key  recommended  tasks. 
For  example,  (1)  in  the  initiating  phase,  it  has  not  defined  general  SPI  goals 
or  guiding  principles;  (2)  in  the  establishing  phase,  it  has  not  developed  an 
SPI  plan  with  measurable  goals  and  committed  resources;  (3)  in  the  acting 
phase,  it  developed  solutions  but  never  pilot-tested  potential  solutions  or 
transitioned  the  solutions  to  long-term  support;  and  (4)  in  the  leveraging 
phase,  it  gathered  lessons  learned  but  did  not  analyze  them  or  use  them  to 
revise  its  organizational  approach.  Further,  it  has  not  reviewed  its 
sponsorship  and  commitment,  established  high-level  goals,  or  decided  to 
continue  with  the  SPI  process.  Without  performing  these  steps,  it  is 
unlikely  that  SPI  activities  will  produce  the  kind  of  meaningful  advances  in 
product  quality  and  cost  savings  that  other  DOD  components  have  realized. 
Table  12  shows  a  detailed  comparison  of  MCTSSA  SPI  activities  against 
each  of  the  IDEALSM  model  recommended  tasks. 


Table  1 1 :  Examples  of  Marine  Corps  Alignment  With  IDEALSM 

Phase/tasks 

Task  example 

Initiating:  Identify  business  needs  and 
drivers  for  improvement 

MCTSSA  identified  the  ever-increasing  amount  of  its  resources  needed  to  support 
software  as  a  business  need  requiring  improvement  and  determined  that  changes  were 
needed  in  its  software  process  and  in  identifying  required  resources. 

Diagnosing:  Plan  for  and  conduct  the 
baseline  of  their  activities 

SEI  performed  a  study  for  the  Corps  that  outlined  program  weaknesses  and  included 
recommendations  for  improvement. 

Establishing:  Finalize  roles  and 
responsibilities  of  infrastructure  entities 

MCTSSA  issued  an  order  that  establishes  the  roles  and  responsibilities  of  SPI 
infrastructure  entities,  including  the  commanding  officer,  technical  adviser,  business 
operations  manager,  division  directors,  and  project  officers. 

Acting:  Pilot-test  potential  solutions 

MCTSSA  pilot-tested  one  project. 

Leveraging:  Analyze  lessons  learned 

MCTSSA  records  lessons  learned  into  a  local  database  that  is  shared  among  Marine 

Corps  divisions. 
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Table  12:  Comparisons  of  Marine  Corps  SPI  Activities  With  the  IDEALSM  Model 

Satisfied? 

Phase 

Task 

MCTSSA 

Initiating 

Organize  discovery  team  to  develop  a  proposal  to  management  for  launching  SPI 
program 

Yes 

Identify  business  needs  and  drivers  for  improvement 

Yes 

Build  an  SPI  proposal 

Yes 

Educate  and  build  support 

Yes 

Obtain  approval  for  SPI  proposal  and  initial  resources 

Yes 

Establish  SPI  infrastructure 

Yes 

Assess  the  climate  for  software  process  improvement 

Yes 

Define  general  SPI  goals 

No 

Define  guiding  principles  of  SPI  program 

No 

Launch  the  program 

Yes 

Diagnosing 

Determine  what  baseline(s)  are  needed 

Yes 

Plan  for  baseline(s) 

Yes 

Conduct  baseline(s) 

Yes 

Present  findings 

Yes 

Develop  final  findings  and  recommendations  report 

Yes 

Communicate  findings  and  recommendations  to  organization 

Yes 

Establishing 

Select  and  get  training  in  a  strategic  planning  process 

Yes 

Review  organization’s  vision 

No 

Review  organization’s  business  plan 

Yes 

Determine  key  business  issues 

No 

Review  past  improvement  efforts 

Yes 

Describe  motivations  to  improve 

Yes 

Identify  current  and  future  (planned)  improvement  efforts 

Yes 

Finalize  roles  and  responsibilities  of  infrastructure  entities 

Yes 

Prioritize  activities  and  develop  improvement  agenda 

Yes 

Reconcile  existing  planned  improvement  efforts  with  baseline  findings  and 
recommendations 

Yes 

Transform  general  SPI  goals  to  measurable  goals 

No 

Create/update  SPI  strategic  plan 

No 

Build  consensus,  review,  approve  SPI  strategic  plan  and  commit  resources 

No 

Form  technical  working  group 

Yes 
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(Continued  From  Previous  Page) 

Satisfied? 

Phase 

Task 

MCTSSA 

Acting 

Complete  tactical  plan  for  technical  working  group 

Yes 

Develop  solutions 

Yes 

Pilot  potential  solutions 

No 

Select  solution  providers 

No 

Determine  long-term  support  needs 

No 

Develop  rollout  strategy  and  plan  template 

No 

Package  improvement  and  turn  over  to  SEPG 

No 

Disband  technical  working  group 

No 

Roll  out  solution 

No 

Transition  to  long-term  support 

No 

Leveraging 

Gather  lessons  learned 

Yes 

Analyze  lessons  learned 

No 

Revise  organizational  approach 

No 

Review  sponsorship  and  commitment 

No 

Establish  high-level  goals 

No 

Develop  new/revised  SPI  proposal 

No 

Continue  with  SPI 

No 

Managing 

Set  the  stage  for  SPI 

Yes 

Organize  the  SPI  program 

Yes 

Plan  the  SPI  program 

Yes 

Staff  the  SPI  program 

Yes 

Monitor  the  SPI  program 

Yes 

Direct  the  SPI  program 

No 

Marine  Corps  Uses  a 
Decentralized  Approach  to 
Manage  SPI  Program 
Activities 


The  Marine  Corps  has  adopted  a  decentralized  management  approach  for 
SPI  by  delegating  responsibility  to  the  Command,  which  in  turn  has 
delegated  most  of  the  responsibility  to  MCTSSA.  Specifically,  the 
Command  retained  overall  responsibility  for  SPI  but  assigned  other 
activities,  such  as  defining  standard  processes  or  metrics,  to  MCTSSA. 
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DFAS’  SPI  Program 


Background  DFAS  provides  finance  and  accounting  services  to  DOD  components. 

Created  in  1991,  it  replaced  more  than  300  service  and  agency  finance  and 
accounting  offices  that  operated  more  than  300  systems.  DFAS  consists  of 
five  centers  and  20  field  offices;  it  operates  83  finance  and  accounting 
systems  but  plans  to  reduce  this  number  to  about  30  or  fewer  by  the  end  of 
2005.  The  systems  are  acquired  and  maintained  by  seven  software/systems 
units  called  systems  engineering  organizations  (SEOs).  (See  figure  7.)  In 
fiscal  year  2000,  DFAS  budgeted  about  $225  million  for  information 
services. 
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Figure  7:  Partial  DFAS  Organization  Chart  Highlighting  Units  Responsible  for 
Information  Systems 


□  DOD  component  or  software/ 
systems  unit  selected  for  review 


Source:  GAO  based  on  DFAS  data. 

DFAS  began  its  SPI  program  in  1993  when  responsibility  for  SPI  was 
assigned  to  the  Financial  Systems  Organization  (FSO)  and  a  corporate 
SEPG  was  established  to  manage  the  program  and  coordinate  SPI  among 
FSO  field  locations.  FSO  established  an  SPI  policy  in  its  1995  SPI  strategic 
action  plan.6  That  policy  is  currently  under  revision.7  The  latest  draft 


6FSO  policy  SM-08. 
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revision  has  continuous  SPI  as  an  objective  for  all  DFAS  SEOs.  The  policy 
also  requires  that  best  practices  be  shared  across  all  DFAS  SEOs  and  that 
process  metrics  be  collected,  maintained,  analyzed,  used,  and  reported  to 
support  the  SPI  process. 


DFAS  SPI  Program  Is 

Aligned  With  SEI’s  IDEALSM 
Model 

DFAS’  SPI  program  is  fully  consistent  with  the  IDEALSM  model.  Table  13 
shows  examples  of  DFAS  activities  that  reflect  some  of  the  recommended 
IDEALSM  tasks;  table  14  shows  a  detailed  comparison  of  DFAS  SPI  activities 
with  each  of  the  IDEALSM  model  recommended  tasks. 

Table  13:  Examples  of  DFAS  Alignment  With  IDEALSM 

Phase/tasks 

Task  example 

Initiating:  Document  the  organization’s  SPI 
approach,  business  needs,  and  team  roles 

DFAS  published  its  original  SPI  action  plan,  which  specified  an  approach,  business  needs, 
and  team  roles,  in  1995. 

Diagnosing:  Plan  for  and  conduct  baselines 

In  1996  DFAS  used  SEI  methods  to  evaluate  process  maturity  at  each  field  location. 

Establishing:  Identify  successful  practices 
to  leverage  and  unsuccessful  practices  to 
avoid 

DFAS  has  a  post-implementation  review  at  one  site  that  uses  staff  input  to  identify  best 
practices  for  each  completed  project;  in  January  2000  it  identified  barriers  to  SPI  success 
and  publicized  these  at  a  DFAS  conference. 

Acting:  Install  solutions  across  the 
organization 

DFAS  completed  a  review  of  current  software  practices  in  2000  and  developed  a  schedule 
to  update  the  current  practices  to  be  consistent  with  CMM®  level  3. 

Leveraging:  Accumulate  lessons  learned 
and  use  them  to  improve  the  software 
process 

DFAS  established  a  corporate  process  asset  library  and  plans  to  link  local  libraries  to  it  so 
information  can  be  shared  across  the  agency.  One  site  has  a  postimplementation  review 
program  that  uses  metrics  to  evaluate  a  project,  analyze  potential  issues,  develop  an 
action  plan,  and  develop  training  to  improve  the  process. 

7DFAS  Regulation  8000. 1-R. 
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Table  14:  Comparisons  of  DFAS  SPI  Activities  With  the  IDEALSM  Model 

Phase 

Task 

Satisfied? 

Initiating 

Organize  discovery  team  to  develop  a  proposal  to  management  for  launching  SPI 
program 

Yes 

Identify  business  needs  and  drivers  for  improvement 

Yes 

Build  an  SPI  proposal 

Yes 

Educate  and  build  support 

Yes 

Obtain  approval  for  SPI  proposal  and  initial  resources 

Yes 

Establish  SPI  infrastructure 

Yes 

Assess  the  climate  for  SPI 

Yes 

Define  general  SPI  goals 

Yes 

Define  guiding  principles  of  SPI  program 

Yes 

Launch  the  program 

Yes 

Diagnosing 

Determine  what  baseline(s)  are  needed 

Yes 

Plan  for  baseline(s) 

Yes 

Conduct  baseline(s) 

Yes 

Present  findings 

Yes 

Develop  final  findings  and  recommendations  report 

Yes 

Communicate  findings  and  recommendations  to  organization 

Yes 

Establishing 

Select  and  get  training  in  a  strategic  planning  process 

Yes 

Review  organization’s  vision 

Yes 

Review  organization’s  business  plan 

Yes 

Determine  key  business  issues 

Yes 

Review  past  improvement  efforts 

Yes 

Describe  motivations  to  improve 

Yes 

Identify  current  and  future  (planned)  improvement  efforts 

Yes 

Finalize  roles  and  responsibilities  of  infrastructure  entities 

Yes 

Prioritize  activities  and  develop  improvement  agenda 

Yes 

Reconcile  existing  planned  improvement  efforts  with  baseline  findings  and 
recommendations 

Yes 

Transform  general  SPI  goals  to  measurable  goals 

Yes 

Create/update  SPI  strategic  plan 

Yes 

Build  consensus,  review,  approve  SPI  strategic  plan  and  commit  resources 

Yes 

Form  technical  working  group 

Yes 
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(Continued  From  Previous  Page) 

Phase 

Task 

Satisfied? 

Acting 

Complete  tactical  plan  for  technical  working  group 

Yes 

Develop  solutions 

Yes 

Pilot  potential  solutions 

Yes 

Select  solution  providers 

Yes 

Determine  long-term  support  needs 

Yes 

Develop  rollout  strategy  and  plan  template 

Yes 

Package  improvement  and  turn  over  to  SEPG 

Yes 

Disband  technical  working  group 

Yes 

Roll  out  solution 

Yes 

Transition  to  long-term  support 

Yes 

Leveraging 

Gather  lessons  learned 

Yes 

Analyze  lessons  learned 

Yes 

Revise  organizational  approach 

Yes 

Review  sponsorship  and  commitment 

Yes 

Establish  high-level  goals 

Yes 

Develop  new/revised  SPI  proposal 

Yes 

Continue  with  SPI 

Yes 

Managing 

Set  the  stage  for  SPI 

Yes 

Organize  the  SPI  program 

Yes 

Plan  the  SPI  program 

Yes 

Staff  the  SPI  program 

Yes 

Monitor  the  SPI  program 

Yes 

Direct  the  SPI  program 

Yes 

DFAS  Reports  That  Its 
Centralized  Approach  to  SPI 
Program  Management  Has 
Produced  Results 


DFAS  centralized  SPI  program  management  when  it  started  its  program 
under  FSO  in  1993.  When  reorganized  in  1998,  DFAS  retained  centralized 
SPI  management  by  assigning  it  to  the  Infrastructure  Services  Organization 
(ISO).  Under  the  draft  SPI  policy  revision,  management  of  the  program  will 
still  be  centralized  in  headquarters,  but  responsibilities  will  be  split 
between  ISO  and  its  parent  unit,  the  Information  and  Technology 
Directorate  (ITD).  Specifically,  the  draft  revision  assigns  the  ITD 
responsibility  for  approving  SPI  policies,  maintaining  metrics,  and 
analyzing  metrics  for  the  purpose  of  recommending  changes  in  priorities, 
resources,  and  processes.  The  draft  revision  assigns  the  ISO  responsibility 
for  publishing  SPI  policies,  maintaining  the  agency  process  assets  library, 
and  coordinating  DFAS-wide  SPI  activities. 
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ITD  reports  that  its  SPI  program  has  improved  DFAS  staff  productivity.  A 
DFAS  contractor  has  conducted  benchmarking  measurements  on  DFAS 
software/systems  development  efforts.  Results  from  1996  and  19978 
measurements  show  that  DFAS  develops  and  maintains  software 
(measured  in  terms  of  function  points9)  for  $0.67  that  costs  an  average 
organization  $1.00,  a  comparable  government  organization  $1.06,  and  a 
comparable  large  commercial  organization  $1.37.  The  contractor  cited 
“strong  processes”  as  one  factor  that  contributed  to  DFAS’  productivity. 


DLA’s  SPI  Program 


Background  DLA  is  a  combat  support  agency  whose  primary  role  is  to  provide  supply 

management,  logistics  services,  and  distribution  support  to  America’s 
military  forces  worldwide.  It  relies  on  software-intensive  systems  to 
administer  over  $900  billion  in  DOD  and  other  agency  contracts.  DLA 
budgeted  about  $784  million  for  information  technology  in  fiscal  year  2000. 

In  1998,  DLA’s  systems  design  center  operated  nine  systems  development 
and  maintenance  units  across  the  country.  After  closing  this  center  in 
December  1998  in  order  to  streamline  operations  and  reduce  costs,  DLA 
created  three  systems  integration  offices  to  oversee  the  development  and 
maintenance  units,  which  have  been  cut  from  nine  to  seven.  (See  figure  8.) 
Each  of  the  three  offices  supports  software  development  and  maintenance 
for  a  separate  DLA  business  function — materiel  management,  logistics,  and 
base  support  and  distribution. 


8Results  from  1998  and  1999  measurements  are  not  yet  available. 

9Function  points  are  software-size  estimates  based  on  the  number  and  complexity  of  inputs, 
outputs,  files,  inquiries,  and  interfaces  for  a  functional  unit  of  software. 
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Figure  8:  Partial  DLA  Organization  Chart  Highlighting  Units  Responsible  for  Software  Systems 


□  DOD  component  or  software/ 
systems  unit  selected  for  review 

Source:  GAO  based  on  DLA  data. 

DLA  does  not  have  an  SPI  program,  having  eliminated  the  program  in  1998 
when  its  system  design  center  was  closed  in  1998.  However,  as  part  of  its 
ongoing  reorganization,  DLA  rewrote  the  policy  and  duties  of  its  CIO  and 
moved  that  function  to  the  new  Information  Operations  unit.  The  CIO  told 
us  that  the  SPI  program  is  to  be  reestablished.  However,  specific  plans  and 
milestones  for  doing  so  were  not  available. 
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Orders  by  Internet: 

For  information  on  how  to  access  GAO  reports  on  the  Internet, 
send  an  e-mail  message  with  “info”  in  the  body  to: 

inf o@  www.  gao .  go  v 

or  visit  GAO’s  World  Wide  Web  home  page  at: 
http  ://www.  gao .  go  v 


Contact  one: 

•  Web  site:  http://www.gao.gov/fraudnet/fraudnet.htm 

•  e-mail:  fraudnet@gao.gov 

•  1-800-424-5454  (automated  answering  system) 
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